Good Indic NLP benchmarks exist. What I could not find was one that evaluates the pipeline people actually deploy: a scanned Hindi or Marathi PDF goes in, OCR happens, the text gets chunked and embedded, retrieval runs, and a model writes an answer with citations. Every existing benchmark starts after the hard part, usually from translated English datasets. Real Indian-language RAG starts from a page image.
So IndicRAG Bench measures the whole path, component by component, across Hindi, Marathi, Tamil, and Gujarati. OCR quality, chunking, embeddings, hybrid retrieval, reranking, grounded generation, and abstention on unanswerable queries, with latency and cost reported next to quality. The constraint I set myself: everything has to run on free resources. Local machine, free GPU tiers, open weights.
The first two weeks are done. Week 1 froze the data schema and config format; every benchmark example carries its license, provenance, supporting spans as character offsets, and an answerable flag, and the schema cannot change without a decision-log entry. The starter set is small and honest about it: 8 native Wikipedia documents and 28 hand-built examples across the four languages, including unanswerable ones.
Week 2 shipped the clean-text retrieval baseline. Two chunkers, one with fixed token windows and one that packs sentences using an Indic-aware splitter I wrote after finding nothing decent off the shelf, feeding a BGE-M3 dense index in FAISS. On the 24 answerable dev queries, structure-aware chunking beat fixed windows at recall@5, 0.979 against 0.951, with the gap sitting mostly in Marathi and Gujarati. A stub lexical embedder on the same config scores 0.833, so the dense model buys roughly 15 points of recall@5 even on this small set.
I want to be clear about what these numbers do not mean. Retrieving 10 chunks from a 21-chunk corpus saturates recall@10 by construction, so only the tighter cutoffs carry signal until the corpus grows. The QA pairs also still need native-speaker verification. Both of those are logged as debts, not footnotes.
Every run hashes its config into its artifact paths, so identical configs never recompute and any number in the results folder can be traced back to the exact command that made it. CI runs a stub end-to-end benchmark on every push.
Next up is the retrieval comparison: two more embedding models, hybrid retrieval with BM25 and reciprocal rank fusion, and the first real failure write-up. The OCR track, the part no other Indic benchmark touches, comes after that.