Ask most assistants a research question and you get a fluent paragraph with no way to check it. That is fine for brainstorming and close to useless for anything you have to defend in a meeting or a written brief. The fix is not a smarter model. It is a stricter pipeline: one that treats every sentence as a claim that must point to a source, and refuses to publish claims it cannot trace. What follows is a build for a research assistant that searches the web or a folder of documents, summarizes what it finds, and hands back citations a skeptical colleague can click, read, and argue with. A working first version is an afternoon of work.
Write the citation contract before you write code
Most projects fail here, not in the retrieval layer. Decide the shape of the output first and treat it as law. Every finding should be a record with a claim, a source id, a short verbatim quote, the source location (URL or file path), a title, and the date you accessed it. The whole answer is then a list of those records plus a short synthesis built only from them.
Two rules make this enforceable. First: no sentence in the final answer may exist without a source id. Second: the quote must be copied, not paraphrased. A paraphrase cannot be checked by a script, and citation accuracy is the one thing you can automate cheaply. Once the contract is fixed, your evaluation stops being a matter of taste.
Pick a retrieval surface and normalize it
You have two realistic options. A document set you already trust, or a web search layer. Start with documents if you have them, because the corpus is stable and you can debug the pipeline without the internet moving under you.
For documents, split into chunks of roughly 400 to 800 tokens with a little overlap, and attach metadata as you go: file path, page or heading, and a stable chunk id. For web retrieval, fetch the full page text rather than relying on the snippets a search engine returns. Snippets are chosen for relevance to a query, not for completeness, and they change between calls. Strip navigation and boilerplate, keep the published date if the page has one, and record your access date.
Make both surfaces speak the same language
Whoever calls retrieval should get back the same object either way: text, source location, title, date, chunk id. Then the rest of your pipeline never needs to know whether a passage came from a PDF or a news page.
Retrieve with provenance, not vibes
Do not send the user's question straight to a search box. Rewrite it into two to four sub-questions, retrieve the top few passages for each, then deduplicate by URL or content hash. This widening step matters more than tuning similarity scores: research questions usually have several parts, and a single query tends to find one of them.
- Rewrite the question into focused sub-queries.
- Retrieve passages for each sub-query and keep the raw text.
- Deduplicate overlapping passages and near-identical pages.
- Persist everything, with ids, so you can re-run the summary without re-fetching.
Store passages in something you can query later, even a local database table. When a citation looks wrong three days from now, you want to see exactly which text the agent saw.
Summarize claim by claim, not paragraph by paragraph
The tempting move is to ask for a nicely written summary and then bolt citations on afterwards. That reliably produces citations attached to the wrong sentence. Reverse the order.
Work in two steps. In the map step, go chunk by chunk and extract candidate claims, each paired with a supporting quote and its chunk id. Keep them short and factual. In the reduce step, group the claims by theme, merge duplicates, and keep the strongest quote for each. Only then write prose, and only from merged, cited claims.
Give the model an explicit escape hatch: if it cannot support a sentence with a chunk id, it should drop the sentence rather than soften it. A shorter answer with four solid citations beats a long one where half the claims evaporate under scrutiny.
Add a verification pass before anything ships
Run a second, separate check over the draft. For each claim, quote, and source, confirm three things: the quote actually appears in the source text, the claim does not overstate the quote, and the link or file still resolves. Normalize whitespace and punctuation before string matching so you are not chasing cosmetic differences.
Overstatement is the interesting failure. A source saying a program reduced errors in one department does not support a claim that it reduces errors across the industry. Flagging scope, numbers, and causal language catches a surprising share of bad citations.
- Verified: quote found, claim matches its scope.
- Weak: quote exists but supports only part of the claim.
- Unsupported: nothing matches, or the source is gone.
Downgrade these in the output instead of silently deleting them. Readers trust a system that flags its own soft spots.
Handle thin and conflicting sources honestly
Sometimes nothing reliable turns up. Make "no source found" a legitimate result with its own formatting, not an error to paper over. When sources disagree — and they will, especially on anything recent — present both positions with attribution rather than averaging them into a mushy middle. Note publication dates next to each claim, because a two-year-old figure and a figure from last month are not the same evidence.
Test with questions you already know the answers to
Before trusting the agent on new questions, build a small set of twenty where you know the correct answer and can name the right sources. Score four things: does the quote exist in the source, does it actually support the claim, did retrieval find the key document, and does the agent say "I couldn't find this" when the answer is genuinely absent. Log every claim-source pair for a week and read a sample by hand. That log is your real quality metric.
Ship the boring version first
Resist adding a second data source, a chat interface, and a scheduling feature in week one. One corpus, one retrieval interface, one output schema, one verification pass. Get citation accuracy high on twenty known questions, then point it at the web. The unglamorous version that a colleague can audit is worth more than a smooth one nobody can check.
Photo: Norman_Gil / Pixabay
