Capture
The browser records the microphone, optionally mixed with audio shared from another tab or app, as WebM/Opus chunks.
MediaRecorder · 1 s chunks
← Back to InterviewMate
Engineering case study
A real-time speech and retrieval pipeline for interview practice
InterviewMate turns spoken practice questions into personalized response suggestions in real time. This page documents the production path: what the system does, where latency budget goes, which safeguards matter, and what has and has not been measured.
Next.js + ReactFastAPI + WebSocketsDeepgram FluxQdrant + Supabase
InterviewMate was first marketed for use during live interviews; it is now positioned for preparation and mock sessions. The pipeline can process any live audio it is given, so the product states the boundary plainly: use it for rehearsal or where AI assistance is explicitly allowed, and disclose it when an organizer requires.
01 / System
The pipeline is a sequence of small latency budgets
Streaming quality comes from keeping each boundary explicit: capture, transcode, turn detection, retrieval, generation, and delivery.
Transcribe
An async FastAPI WebSocket pipes chunks through an FFmpeg subprocess to 16 kHz linear PCM and streams them to Deepgram.
FFmpeg · WebSocket · Deepgram flux-general-en
Detect
Deepgram end-of-turn events and lexical checks mark likely questions; low-confidence short utterances are verified by a model call.
EOT threshold 0.7 · 800 ms EOT timeout
Retrieve
Compound questions are split into up to three sub-queries; each searches only the user’s own prepared Q&A pairs.
Qdrant user_id filter · 5 s per-search timeout
Respond
A closely matching prepared answer, or a streamed model response, is sent back over the same socket.
Prepared-answer match · Claude streaming · prompt caching
02 / Latency
Latency is a budget, not a slogan
These are the configured bounds in the code, not measured end-to-end numbers. Turn detection alone spends most of a second by design, so first-token and complete-response latency need separate measurement rather than one headline figure.
| Stage | Bound | Why |
|---|---|---|
| Client chunk interval | 1,000 ms | Set in the practice page recorder; trades socket chatter for turn latency |
| End-of-turn timeout | 800 ms | Deepgram EOT timeout; server-side end-of-turn events decide when a question is complete |
| Retrieval bound | 5 s per sub-query | Up to three parallel searches; a timeout yields partial context, not a stalled turn |
03 / Retrieval
User-specific retrieval keeps context useful and isolated
Prepared Q&A pairs are embedded and searched semantically. Vector searches are filtered by the authenticated user ID, so a faster answer is not allowed to come at the cost of another user's context.
- Direct match: a prepared answer at ≥0.85 similarity is returned without a generation call. The threshold was raised from 0.70 after a wrong prepared answer surfaced in production.
- Compound question: retrieve several relevant pairs, then synthesize one response while tracking examples already used in the session.
- Graceful degradation: if Qdrant is unavailable, the model is given the user's own prepared Q&A pairs directly; a search timeout returns partial context rather than blocking the turn.
Cache layers
1
Prepared-answer match
Exact and lexical similarity (max of substring, token Jaccard, and sequence ratio) against the user's own prepared Q&A. At 0.85 or above the prepared answer is returned without a generation call.
2
Semantic retrieval
Qdrant finds paraphrases and related prepared answers inside the current user scope.
3
Prompt caching
The stable system prompt is marked for Anthropic prompt caching so repeated turns avoid re-processing it.
04 / Routing
Routing and fallback protect the interaction
Question detection
Fast pattern checks handle obvious turns. Low-confidence cases can fall back to model verification instead of blocking every turn.
Model choice
A lower-cost GLM-first hybrid was tried and turned off in February 2026 because it ignored the user profile, prepared Q&A, and session context. Claude is the only answer model.
Operational bounds
Timeouts, partial retrieval, connection cleanup, and user-scoped filters keep a failed search from stalling the turn.
05 / Evaluation
Evaluation methodology: measure quality and failure, not just speed
The useful unit is a question-to-response turn. The repository contains benchmark scripts for question detection, Q&A cache lookup, and model latency, plus reproducible prompt experiments with raw outputs and summaries. What it does not yet have is an automated regression suite or committed production latency data.
Component benchmarks
Scripts time regex question detection, cache lookup, and GLM vs Claude responses. They print results; they do not assert thresholds.
Controlled prompt runs
Fixed question, fixed model and temperature, 20–100 runs per condition, automatic pass/fail scoring with ambiguous cases kept separate.
Production A/B signal
Each user is assigned a prompt variant through Statsig; thumbs up/down on suggestions is logged against that variant. Session transcripts can be exported for review.
Known gap
No automated regression suite, committed production latency data, or independently audited SLO yet.
06 / Research
A lucky answer became a controlled experiment
A widely shared prompt (“The car wash is 50 meters away. Should I walk or drive?”) trips many models because the car must be at the car wash. In one practice session InterviewMate answered “drive”. We did not know which prompt layer caused it, so we isolated them.
Ablation
Six prompt conditions, 20 runs each, claude-sonnet-4-5 at temperature 0.7. A short role + STAR scaffold passed 17/20; role + profile context passed 6/20; bare and role-only passed 0/20.
Reproduction on production
The same STAR scaffold inside the full production prompt passed 0/20 and 6/20 depending on profile. Standalone it passed 20/20, then 100/100 on claude-sonnet-4-6. The earlier production “drive” used distance-based reasoning. It was right for the wrong reason.
What changed
The investigation also found a similarity bug that returned 0.95 for unrelated equal-length strings and a too-loose 0.70 retrieval threshold. Both were fixed. New profiles now default to the short STAR prompt, and session scenario hints are added to the user turn instead of the system prompt.
Limits: one question, small samples, Anthropic models only in these runs. The result supports a narrow claim, that instruction ordering in a long prompt can suppress a reasoning scaffold. It does not show that any prompt works for every topic. Code, raw outputs, and summaries.
Reliability notes
Incidents became architecture decisions
Observed failureChange madeLesson
Observed failure
Thread bridge created 5-second stalls
Change made
Moved audio forwarding to a fully async subprocess and direct await.
Lesson
Keep the hot path on one async event loop; avoid blocking waits around streaming I/O.
Observed failure
Long questions could hang during decomposition
Change made
Replaced a model round-trip for decomposition with heuristic splitting (max three sub-queries) and added 5-second per-search timeouts.
Lesson
A useful partial result is better than an unbounded wait in a real-time turn.
Observed failure
RAG was silently bypassed
Change made
Replaced an unconfigured global service with lazy Supabase-aware service construction.
Lesson
Dependency wiring is part of retrieval correctness, not just application plumbing.
Observed failure
Early Deepgram streaming was timeout-prone
Change made
Moved to Flux-style end-of-turn detection with eager thresholds and explicit cleanup.
Lesson
Turn detection and resource cleanup deserve first-class reliability tests.