Seedling note
Rough thoughts. Expect this to change a lot.
Things to measure separately:
- Retrieval: did we fetch the right chunks? (recall@k, MRR)
- Grounding: is the answer supported by those chunks?
- Usefulness: did it answer the actual question?
Ideas to explore:
- Build a small golden set of 50 questions
- Compare chunk sizes 256 / 512 / 1024
- Log every retrieval with its scores
Related: Agentic systems