This one started as Task 2 of the HH Goa 2026 shortlisting, which we entered as a team. The pipeline in this post is the part I built; a teammate shipped the hosted link. The brief: someone speaks a question, the system transcribes it, finds the right passages and answers from them. The corpus was the Hindi validation slice of MSMARCO-XI, which is MS MARCO machine-translated into Indic languages by AI4Bharat. That slice holds 97,941 queries and 950,721 unique passages, and every row keeps the English and Hindi text side by side.
The brief set one number: everything after the transcript, meaning the guards, embedding, retrieval, abstention and the answer itself, had to finish in under 200 ms. I held that at P90. Transcription isn't in the budget. The design streams Sarvam's realtime speech-to-text and fires retrieval on partial transcripts, so the index can work while the person is still talking. The voice path that's verified end to end today is a recorded clip sent to Sarvam's REST endpoint, though. The realtime session stays switched off until one has been confirmed against Sarvam's current protocol.
Retrieval took the most testing. I built three strategies and ran them head to head on the full corpus: passage-level dense with multilingual-e5-small in a usearch HNSW index, sentence-level dense that maps back to the parent passage, and BM25 over the English side. Passage dense plus BM25, fused with reciprocal rank fusion, won. Sentence-level lost, which makes sense for passages this short: they average 317 characters. I kept the losing config in the repo anyway, because the comparison is the useful part. BM25 also needed work of its own: the first scorer took 145 ms across 950K documents, and a numba version brought that under 1 ms.
Generation runs Qwen3-0.6B on vLLM, on one Blackwell MIG slice with 24 GB. Speculative decoding paid off more than I expected. RAG answers mostly copy spans straight out of the retrieved context, which is the best case for prompt lookup, so English generation p50 fell from 84.7 ms to 43.5 ms and Hindi from 242 ms to 131 ms. The answers came out byte-identical. On my laptop's RTX 4060 with llama.cpp the same model still generates an answer in 77 ms.
The measured numbers, on 1,000 real queries with generation and guards switched on:
- English P50 122.0 ms, P90 169.5 ms
- Hindi P50 102.9 ms, P90 188.5 ms
- English MRR@10 0.4066 and recall@10 0.7417 on answerable queries, against 0.1943 and 0.3428 for Hindi text queries
P100 is 357 ms in English and 530 ms in Hindi. That's two outliers out of a thousand, and the reports show them instead of trimming them.
The guards were the honest part to get right. Roughly 45% of MSMARCO queries have no answer in the corpus, so abstaining is a real skill here, not an edge case. There are four gates: a prompt-injection classifier on input, a retrieval-confidence gate, the model's own NO_ANSWER contract and an HHEM groundedness check on the answer. Two of them were only trained on English. Rather than pretend otherwise, they step aside for Devanagari text and say so in the trace, and an answer that couldn't be checked gets marked unchecked instead of quietly passing. If generation blows its deadline, the pipeline falls back to an extractive answer in under 10 ms rather than blowing the budget.
Every stage runs inside a small harness with timeouts, retries and fallbacks, and each request carries a per-stage trace. So the latency table comes from the same code path the demo runs, not from a benchmark script on the side. Codex reviewed the plans across three rounds, and every finding went into the plan docs with what was accepted, what was deferred and why.
What I'd take from it: a budget forces design decisions early. Once 200 ms was fixed, things like starting retrieval on partial transcripts and keeping a model-free answer floor stopped being nice-to-haves.