RAG Pipeline with Eval Harness

What you're building

Pick a corpus you actually care about — Pacvue docs, a stack of technical PDFs, or a large OSS project's documentation. Two entry points: an ingest command that chunks, embeds, and indexes the corpus into local Qdrant, and an ask command that retrieves (any of four toggleable configs), optionally reranks, and generates a cited answer. Around them, an eval harness: ≥50 human-verified question/answer/source triples, and a script that emits the full metrics table as a markdown report you commit to the repo.

Suggested structure

# config.py — every stage toggleable; the eval report depends on this
CONFIG = {
    "chunk_size": 350, "chunk_overlap": 50,
    "use_dense": True, "use_bm25": True,     # hybrid = both on
    "use_rerank": True,
    "retrieve_k": 50, "final_k": 5,
}

# ingest.py
def ingest(corpus_dir: str) -> None:
    # 1. parse each file to clean text (handle at least .md and .pdf)
    # 2. structural_chunks() with CONFIG sizes; keep doc_id/heading/position
    # 3. batch-encode embed_text; upsert to Qdrant with payload metadata
    # 4. build + pickle the BM25 index over the same chunk list
    ...

# pipeline.py
def retrieve(query: str, cfg: dict) -> list[int]:
    rankings = []
    if cfg["use_dense"]:
        rankings.append(dense_search(query, cfg["retrieve_k"]))
    if cfg["use_bm25"]:
        rankings.append(bm25_search(query, cfg["retrieve_k"]))
    ids = rrf_fuse(rankings) if len(rankings) > 1 else rankings[0]
    if cfg["use_rerank"]:
        ids = rerank(query, ids[:cfg["retrieve_k"]], top=cfg["final_k"])
    return ids[:cfg["final_k"]]

def ask(query: str, cfg: dict) -> dict:
    # grounded generation with numbered citations + explicit refusal path
    ...

# evaluate.py
def run_eval(eval_set_path: str, report_path: str) -> None:
    # for each config (dense/bm25/hybrid/hybrid+rerank):
    #   precision@5, recall@5, MRR over the eval set
    # for the best config: faithfulness + answer relevance (RAGAS or
    # your own LLM-as-judge — claude-sonnet-5 or gpt-5.5)
    # write eval_report.md: metrics table + 'what I'd improve' section
    ...

The eval report is the deliverable. The committed eval_report.md is what separates this from every tutorial RAG repo: a table of precision@5 / recall@5 / MRR for dense-only vs. BM25-only vs. hybrid vs. hybrid+rerank, faithfulness and answer relevance for the best config, and an honest "what I'd improve" section. Your practical test is presenting this report to Claude as if to a hiring panel — be ready to defend the chunking parameters and fusion choice with your own numbers, not folklore.

Ship it to your portfolio

• README with a 60-second demo: one architecture diagram, one ingest + ask command pair with real output (a cited answer), and a short GIF or asciinema recording. A reviewer should see the pipeline work before reading a single paragraph.
• Eval report front and center: link eval_report.md from the top of the README and surface the headline numbers there — retrieval precision@5/recall@5 per config and faithfulness for the best one. The metrics table is the project; don't bury it below setup instructions.
• An honest "Limitations" section: what the eval set doesn't cover (multi-hop? multilingual? table-heavy PDFs?), where the pipeline still fails, and what you'd build next. Candid limitations read as seniority; a repo claiming perfection reads as a tutorial clone.
• Hiring managers check GitHub before the résumé — pin this repo, make the description one crisp sentence with your best metric in it, and keep the commit history readable. One deep, evaluated project beats five shallow demos.