Independent open-source project · Active development
Modular RAG Engineering, Evaluation & Observability.
An open-source, Python-based RAG system I designed and implemented, covering ingestion, indexing, retrieval, configurable inference, structured logging, testing, and benchmark-driven evaluation.
Problem
Connecting a language model to documents is not enough to understand whether a knowledge system is finding the evidence needed to answer a question. The project explores how to make document processing, retrieval behavior, evidence grounding, and failure analysis more measurable and maintainable.
Contribution
I designed and implemented an independent, modular RAG system. The implementation separates document ingestion, retrieval, language-model routing, logging, and evaluation, so changes can be investigated through component-level evidence.
It is an independent engineering project, not a claimed commercial or production deployment.
Architecture & data flow
Documents into searchable evidence
- PDFs are loaded.
- Text is cleaned and chunked using configurable rules.
- Hugging Face embeddings are created.
- Vectors persist in Chroma.
Retrieve, ground, generate
- Embed the question and search the vector store.
- Use MMR and exact-text de-duplication to select context.
- Build a bounded context and grounded prompt.
- A unified client routes to Ollama (local) or Gemini (remote).
- Return an answer with model metadata.
Inspect evidence and failures
- Compare retrieved chunks with labeled supporting facts.
- Apply deterministic lexical matching.
- Report Hit Rate@K, Recall@K, and MRR.
- Keep JSONL traces, latency/model metadata, and readable reports.
docs/diagrams/llm-systems-architecture.mmd.The code and documentation support PDF loading, text cleaning, configurable recursive chunking, Hugging Face embeddings, persistent Chroma storage, semantic search, MMR-based candidate selection, exact-text de-duplication, and bounded context assembly. A unified client routes generation to local Ollama or remote Gemini, based on configuration.
Evaluation & testing
Evaluation uses labeled questions and structured supporting facts. A deterministic lexical matcher checks whether the required text spans appear in retrieved context. The framework reports Hit Rate@K, Recall@K, and MRR and can generate sample-level reports for failure investigation. Repository unit tests cover retrieval metrics, reports, and JSONL logging.
In this project, Hit Rate@K is a strict answerability-style measure: all required facts for a sample must be present in the top-k context. Recall@K measures the fraction of supporting facts found, while MRR records how early the retrieved context becomes sufficient.
Observability & engineering
Structured JSONL events preserve retrieval traces, evaluation-run details, latency, and model metadata. The repository also includes configuration and architecture documentation, a Python project manifest and lockfile, and focused unit tests. These choices support reproducible debugging and a measured improvement loop.
Limitations & next steps
The evaluator is a deterministic lexical baseline. It does not recognize paraphrases or semantic equivalence, and it can satisfy a multi-span fact using text found across unrelated chunks because locality is not enforced. Those limits are documented in the repository.
Knowledge-graph construction, entity and temporal modeling, graph retrieval, agent workflows/tool use, and semantic evaluation are described as future directions—not implemented features. Planned provider support beyond Ollama and Gemini is also not presented as available functionality.