I Automated Our Weekly Report With AI — Here Is What Broke First

The pitch for automating a weekly report was simple: pull the week's numbers from our project tracker, hand them to an LLM, get back a readable summary instead of someone manually writing the same three paragraphs every Friday. The first version worked well enough to nearly ship as-is, and then it confidently stated a completion percentage that didn't match any number in the actual source data, in a sentence that read exactly as trustworthy as the sentences next to it that were correct.

The first version, and why it looked fine

The pipeline pulled a JSON export from our project tracker — tickets closed, tickets opened, average cycle time — and handed it to an LLM with a loose prompt:

Write a friendly weekly status summary based on this data:

{$jsonData}

Keep it to 3-4 sentences, upbeat but honest tone.

For the first few weeks, the output read well. It was only in week four, when someone cross-checked the summary against the raw dashboard before sending it out, that a sentence claiming "we closed 87% of open tickets this week" turned out to correspond to no actual number in the source data at all — the real figure was 71%. The model hadn't misread a number; there's a strong chance it generated a plausible-sounding percentage in the general shape a status report expects, because nothing in the prompt constrained it to only state numbers that existed in the input.

Why a loose prompt invites this specific failure

"Write a friendly summary" is an instruction about tone and structure, not about the one thing that actually mattered here: every single number in the output has to trace back to a number that was actually in the input. Nothing in the original prompt said that explicitly, so the model optimized for what it was actually asked for — a summary that reads like a good status report — and a good status report, in the training data this kind of model has seen, usually contains a completion percentage. The model filled that expectation in a stylistically plausible way rather than a factually grounded one, because factual grounding was never actually specified as a requirement.

The fix: constrain the output format so every number is traceable

Write a weekly status summary using ONLY the numbers provided below.
Do not calculate, estimate, or restate any number that is not
explicitly present in this data. If you want to describe a trend
or percentage, only do so if it is already present as a field
below — do not compute a new one.

Data:
- tickets_closed: {$data['tickets_closed']}
- tickets_opened: {$data['tickets_opened']}
- avg_cycle_time_days: {$data['avg_cycle_time_days']}
- completion_rate_percent: {$data['completion_rate_percent']}

Format: exactly 3 sentences. Sentence 1: tickets closed vs opened.
Sentence 2: cycle time, compared to last week's value of
{$data['last_week_cycle_time']} if different. Sentence 3: one
observation about the completion rate, using only the number
provided.

The critical change is "do not calculate... any number that is not explicitly present" combined with pre-computing every number we actually wanted mentioned (including completion_rate_percent itself) before the prompt ever runs, rather than asking the model to compute or estimate anything from raw counts. If a percentage matters, calculate it in PHP and hand it over as a labeled field — never ask the model to do arithmetic on your source data and trust the result blindly.

A validation step that catches the next version of this bug

Constraining the prompt reduced the problem but didn't eliminate the need to check for it, so we added an automated post-generation check: extract every number that appears in the generated text and confirm each one appears somewhere in the source data it was supposed to be built from.

function validateNumbersAreGrounded(string $generatedText, array $sourceData): array
{
    preg_match_all('/\d+(?:\.\d+)?%?/', $generatedText, $matches);
    $numbersInText = $matches[0];
    $sourceValues = array_map('strval', array_values($sourceData));

    $unverified = array_filter($numbersInText, function ($number) use ($sourceValues) {
        $clean = rtrim($number, '%');
        return ! in_array($clean, $sourceValues, true);
    });

    return array_values($unverified);
}

$unverified = validateNumbersAreGrounded($summary, $data);

if (! empty($unverified)) {
    // Flag for human review instead of auto-sending — a number
    // showed up that we can't trace back to the source data
    Log::warning('Ungrounded numbers in AI report', ['numbers' => $unverified]);
}

This is a blunt instrument — a plain regex extracting anything number-shaped — and it's caught real problems since, including a case where the model restated a cycle-time number rounded differently than the source (4.5 days became "about 5 days," which is defensible in prose but technically didn't match the exact source value, and the check correctly flagged it for a human to glance at rather than silently letting it through).

What we learned applies beyond this one report

• Never ask a model to do arithmetic on source data you care about. Compute it yourself, hand over the result as a labeled field, and constrain the model to describing it, not deriving it.
• "Only use the provided data" needs to be explicit and repeated, not implied by handing over data and hoping the model infers the constraint on its own.
• A cheap automated check beats a one-time manual proofread. The human catch that found the original bug was a lucky spot-check, not a process — the regex validator above runs on every single report, every week, without anyone needing to remember to look.
• Report structure should be as rigid as the situation allows. A fixed sentence-by-sentence format, each tied to a specific labeled data field, leaves far less room for the model to fill gaps with plausible-sounding invention than an open-ended "write a summary" instruction does.

A second, quieter failure: tone drift over successive weeks

Once the numbers problem was fixed, a subtler issue showed up over the following month: the "upbeat but honest" tone instruction, left to the model's own interpretation week over week, gradually drifted toward more enthusiastic phrasing than the actual numbers supported. A week where cycle time got slightly worse got described as "a small adjustment period" rather than plainly stated as a regression, because "upbeat" is a vague enough instruction that the model's idea of upbeat crept further from neutral each week, with nothing anchoring it back to a fixed baseline.

The fix was replacing "upbeat but honest" with a concrete rule: describe any week-over-week change using neutral, specific comparative language ("cycle time increased by 0.3 days compared to last week") rather than a subjective adjective, and reserve any positive framing for cases where the underlying number actually improved. Vague tone instructions compound their own drift because there's no ground truth to check them against; a rule tied directly to whether a number went up or down doesn't have anywhere to drift.

Deciding what still needs a human before it goes out

Even with grounding and validation in place, the report still goes to a human for a quick read before it's sent, not because we don't trust the numbers anymore, but because a report can be numerically accurate and still miss context a human would catch — a genuinely bad week that happened to coincide with a planned team vacation, for instance, where the raw numbers alone would read as a concerning dip without that context attached. The validation step catches factual errors. It was never meant to replace someone glancing at the report before it reaches a wider audience, and treating an automated check as a full replacement for that glance, rather than a filter that makes the glance faster, is a mistake worth naming explicitly rather than assuming automation implies zero oversight.

• Automated check catches: numbers not present in the source data, obvious factual mismatches.
• Automated check misses: numbers that are technically accurate but presented without context a human reader would want, like the vacation-week example above.
• The review step stays lightweight on purpose — a glance, not a rewrite — because making it heavy again would defeat the entire point of automating the first draft.

Where this pipeline lives now

The actual data-pulling and prompt-assembly steps run as a scheduled job, sketched out originally using the AI Automation Builder to lay out the trigger, the data-fetch step, and the review-before-send gate before any of it was wired into real code — the same "map it out in plain English first" approach that made the review-step requirement visible early instead of getting bolted on after the hallucination bug already happened once. The pattern of generating a draft, constraining what it's allowed to state, and adding a human or automated review gate before anything goes out is the same shape as the support ticket triage workflow covered separately — draft, don't auto-send, and build the validation for the specific way that particular pipeline is most likely to quietly go wrong.

What changed about how we write prompts after this

The lasting change from this whole episode wasn't the report specifically, it was a habit that carried into every other automation we've built since: any time a task involves a model describing or summarizing numeric data, the numbers themselves get computed in code first and handed over as labeled, pre-calculated fields, never as raw data the model is trusted to read, interpret, and restate correctly on its own. That single rule would have prevented the original bug outright, and it's cheap enough to apply by default now rather than something we have to remember to think about case by case.

The other lasting change is smaller but just as practical: every new AI-generated output that goes anywhere outside the team who built it now gets at least one explicit, named validation check before it ships, even if that check is as blunt as the number-matching regex above. Not because every task needs elaborate grounding infrastructure, but because writing down "here's specifically how this could quietly be wrong, and here's the cheap check for it" during development is a lot cheaper than discovering the failure mode in production after someone downstream has already acted on bad information.