← Back to projects

Project Case Study

CallQA

Evaluation-first pipeline for support call intelligence that scores transcription, diarization, event detection, and summaries against ground truth instead of just producing them.

  • 16.9% WER with Whisper small.en on real telephone audio, 1 to 2% on the synthetic tier.
  • Call-level F1 of 1.00 on silence, escalation, and compliance detection in distribution. Interruption scored 0.143, and the diagnostic explaining why is in the repo.
  • Six-field agent scorecard with evidence timestamps, summaries scored with ROUGE plus an LLM grounding judge, all behind 369 offline tests.
GitHub Repository

Context

Every call analytics pitch shows the same thing: audio goes in, a transcript and a summary come out, everyone nods. What none of them show is an error bar. If you are going to trust a pipeline to flag compliance misses or score agents, you need to know where it breaks before it breaks on a real call.

Problem

Measuring a speech pipeline needs ground truth, and real support calls do not come with any. Human annotation is slow and fuzzy at the timestamp level, and real customer audio drags privacy problems in with it. So the question became: how do you get labels you can fully trust without touching a single real customer call?

Approach

The dataset has two tiers. The core is synthetic: short support calls written by an LLM, spoken by Kokoro TTS, and assembled on a timeline, so every silence, interruption, escalation, and compliance label is exact by construction. That audio then passes through telephone-band degradation, otherwise the scores would be fantasy. The anchor is 40 calls from the public HarperValley Bank corpus, which keeps the WER and diarization numbers honest on real telephone audio.

On top of that sits the pipeline itself: a local Whisper sweep for ASR with cloud adapters gated behind optional keys, pyannote for diarization, four event detectors, a six-field agent scorecard with evidence timestamps, and distilbart summaries checked with ROUGE plus an LLM judge for grounding. A Streamlit dashboard lets you open any call and see the gold and predicted speaker turns side by side, and a single-wav CLI runs the whole pipeline on any audio file you hand it.

Outcome

The numbers came out mixed, which is the point. ASR landed at 16.9% WER on the real tier with small.en. The event detectors hit call-level F1 of 1.00 on silence, escalation, and compliance in distribution, against labels that are exact by construction. Diarization was the honest disappointment: pyannote over-splits short telephone calls into 4 to 7 speakers when the gold says 2, and the dashboard timeline shows it instead of burying it in a pooled DER of 0.362.

The best story is the failure. Interruption detection scored F1 0.143. A frame-level diagnostic traced it to clip-edge overlaps with weak voiced content in the synthetic assembly, and a barge-in variant validated the fix at 0.667 without touching the canonical set. Cloud provider rows stay pending until a key lands; nothing is faked. The whole suite runs offline, 369 tests, no GPU or network needed.

Next Iteration

  • Land Deepgram and AssemblyAI keys so the cost-per-audio-hour comparison fills in with real rows.
  • Grow the out-of-distribution holdouts; one escalation OOD read (0.667) is not enough to claim generalization.