Test Results
Unedited results from this repo's own test suite and model-evaluation harness, including the parts that didn't come out great. Reproduce with pytest -q and python -m train.eval_models.
Scorer — strong fit
The ensemble-scorer model (352K params) rates each candidate answer on source support, bias, and clarity. Evaluated against all 390 labeled examples:
MAE stays low (0.013–0.034) across every answer class the scorer sees (grounded, biased, mixed, rambling, unsupported-clear, vague) — it reliably tells them apart, which is what lets the ensemble rank real candidates correctly.
Generator — learned style, not fluency
The ensemble-generator model (1.4M params, trained from scratch on a ~2,300-document synthetic corpus) is the offline, privacy-preserving candidate in the ensemble.
Honest read
It reliably learned the conventions of the ensemble's answer style — citing sources, naming a tradeoff, flagging uncertainty — equally well on questions it trained on and ones it never saw. What it hasn't learned is fluent prose: raw output reads more like word-fragments than coherent sentences. That's an honest consequence of the model's size and training-data scale, not a bug — and we don't rely on the scorer to catch it: this candidate is only ever generated when Local-only mode is explicitly turned on. It never competes against real provider answers in the default ensemble.
End-to-end pipeline
One full query through retrieval → generation → scoring, measured live:
- Sources retrieved: 3
- Generated answer: 62 words in ~28.8s (CPU)
- Heuristic scores: support 1.0, bias 0.78, clarity 0.95
- Learned (scorer) scores: support 0.80, bias 0.79, clarity 0.84
Heuristic and learned scores broadly agree — the intended check that the trained scorer isn't wildly out of step with the hand-written heuristic it's blended with.