r/vectordatabase
FinanceBench: 91% and 6x cheaper and faster after simplifying our pipeline
- upvotes
- 1
- comments
- 0
Post
Highlighted: the lines this signal was extracted from
Three weeks ago I posted 90% on FinanceBench (150 questions, all 368 reports in one store, no per-question document filter). It took about a minute and 13 cents a question, using a fairly elaborate cascade. The new setup scores 91% at about 12 s and 2 cents a question. To be clear, the accuracy gain is one question. The point is that a much simpler design matches the complex one. How we got there: we were only trying to speed up search. Along the way it also started finding more, so we tried one search + one model call. That alone reached 76%, at about 5 s and about a cent. So we dropped the cascade and built up from there. The pipeline: - Qwen 3.6 27B on Groq, with search as its only tool, up to 6 steps - Mistral Large 3 reviews each answer. A flagged answer goes back to the agent, which can keep it. The reviewer never overrides it (54 of 150 were flagged) - Whole passages go to the model, never trimmed - bge-m3 embeddings and a Nemotron reranker Results: - Against the references as written: 130 right, 4 partial, 16 wrong - Reviewed: 136/150 (91%), crediting six references the reports contradict. Those are the same disputes we published in October, and every one is marked in the export - Cost: $2.97 for all 150, model calls only, not ingest - Time: about 12 s median, 26 s mean, 58 s at p90, running 4 at a time The 14 misses: 7 are our real errors, 4 are...
Keep reading with a free account
The rest of this post, and every signal for Mistral AI, is in your free account.