Why Your AI Assistant Gives Wrong Answers About Your Own Project: An Ingestion Post-Mortem

The model is usually not the first suspect

Your AI assistant gives wrong answers about your own project because the knowledge base often feeds it the wrong evidence. The model may sound confident, but the failure usually starts earlier: during ingestion, parsing, chunking, versioning, tagging, and ranking. If the assistant cites a draft instead of the signed version, or reads a budget table as random text, better prompting will not fix the root cause.

This is the post-mortem pattern we see when operating source-grounded assistants for project teams and communities. A consortium uploads proposals, grant agreements, meeting notes, deliverables, reports, and guidance. The assistant answers with citations, but one answer is off. The team blames the model, then the logs show a simpler story: the retriever found a badly ingested source and treated it as authoritative.

A retrieval assistant is only as reliable as its document pipeline. The practical question is not “which model did we use?” It is: which documents were allowed into the index, how were they split, which version won, and what metadata told the assistant what to trust.

The ingestion defects that cause confident wrong answers

The most common defects are boring, which is why they survive. They sit inside file names, PDF structure, export settings, and folder habits.

• Mangled tables. One project uploaded a signed budget annex as a scanned PDF. The extraction joined column headings from one page with values from another. The assistant answered with a plausible staff cost because the text contained the right words, but the row belonged to another work package.
• Duplicate versions. A folder contained “final”, “final revised”, and “final signed”. The assistant cited “final revised” because it had cleaner text and ranked higher. The signed version was present, but had no authority marker.
• Untagged drafts. A concept note described activities that were later removed. It was indexed beside the approved application with no draft label. The assistant treated the removed activity as still planned.
• Section-blind chunking. A deliverable template had headings, annexes, and instructions. The chunker cut through a section boundary, so the assistant mixed guidance text with project commitments.
• Orphaned files. A meeting note referenced “the agreed KPI list”, but the actual list lived in a spreadsheet stored elsewhere. The assistant found the note, not the source list.

Each defect produces the same user experience: the answer looks sourced, but the source is not the source you meant.

Fix provenance before you tune prompts

Prompt tuning can reduce style problems. It cannot decide that one file is legally binding and another is an old working draft unless the index carries that information. Trustworthy citations need metadata that travels with every chunk.

Start with provenance. Every chunk should know the original file, the upload route, the source system, and the person or role that supplied it. A citation to “annex.pdf” is weak. A citation to the signed grant agreement annex uploaded from the coordinator workspace is stronger.

Add dates in more than one form. The file modification date is not enough. Store the document date, approval date, upload date, and validity period when you have them. A newer upload can contain older content. An older signed agreement can outrank a newer comment draft.

Define authority tiers. A simple tier set is often enough: signed agreement, approved deliverable, official guidance, internal decision, draft, template, archive. The retriever can then prefer higher-authority material when two chunks answer the same question.

Track relationships. A deliverable may supersede a draft. A board decision may approve a change request. A guidance note may apply only to one call or programme strand. These links stop the assistant from flattening your knowledge base into a pile of equally valid text.

Structure the knowledge base around decisions, not folders

Many project folders mirror how people worked, not how an assistant must answer. They contain inboxes, exports, partner folders, old versions, and copied attachments. That structure is useful for humans during delivery, but weak for retrieval.

A better structure starts from the questions the assistant must answer. For an EU cooperation project, those questions usually cluster around commitments, eligibility, roles, events, deliverables, indicators, reporting, governance, and evidence. Build collections around those domains, then map documents into them with metadata.

The ingestion pipeline should preserve headings, page numbers, tables, and annex boundaries. It should also label boilerplate, instructions, signatures, comments, and superseded sections. When the assistant quotes a deliverable, the citation should point to the section that contains the commitment, not to a random chunk from the same file.

Use a review queue for risky content. Scans, spreadsheets, merged PDFs, templates, and files with “draft” in the name should not quietly enter the trusted index. They need extraction checks and authority labels. If a project has sensitive internal notes, decide whether the assistant may use them for answers, only for internal search, or not at all.

This is the less glamorous part of AI work, but it is where most accuracy gains come from.

A practical ingestion checklist for project teams

Run this checklist before you judge the assistant. It will find defects that model benchmarks miss.

• List source systems. Name the places documents come from: shared drive, project platform, email export, website, survey tool, event tool, finance folder.
• Separate document classes. Keep contracts, approved applications, deliverables, guidance, minutes, drafts, templates, and raw evidence distinct in metadata.
• Declare authority. Give each class a default authority tier, then allow exceptions for specific files.
• Resolve versions. Mark superseded files. Do not rely on file names to identify the latest approved document.
• Test extraction. Open parsed text for PDFs, tables, and spreadsheets. Check whether rows, columns, footnotes, and page headers survived.
• Chunk by structure. Split by headings, sections, annexes, and table blocks, not by a fixed character count alone.
• Attach dates. Store document date, approval date, upload date, and validity period where relevant.
• Require citation previews. Let editors see the exact snippet the assistant will cite.
• Log misses. When users correct an answer, record whether the issue was missing content, bad ranking, wrong authority, or broken extraction.

If you want help turning this into a working system, StrandsUnited designs, builds, and operates knowledge platforms and source-grounded assistants for cooperation projects and communities. Our work spans ingestion pipelines, member areas, catalogues, and AI assistants that answer with sources, described at strandsunited.com/platforms.

Where this comes from

We see these patterns while operating Impactful, building knowledge systems for communities, and maintaining our grant intelligence engine with a consortium graph from CORDIS and Erasmus+ open data. The same lesson holds across project platforms and internal assistants: citations become trustworthy when ingestion carries provenance, dates, structure, and authority. Start by auditing your indexed documents, then fix the pipeline before asking the model to be wiser.