Something has gone wrong with LLM summarization pipelines. The standard advice in the WordPress ecosystem has settled on “just prompt it harder,” and data integrity is paying for it. Fourteen years of messy databases and broken checkouts taught me that a data layer built on a guess will eventually take the application down with it.
I keep running into summarizers that take a five-minute meeting transcript and produce eight tidy sections. The output looks professional. It reads like somebody was paying attention. Then you check it against the transcript and half the action items were invented out of one ambiguous sentence. You cannot prompt your way out of that, because the pipeline is what let it through.
The identification step most pipelines skip
Causal inference has a hard rule: identification comes before estimation. Identification is the argument that your data can support the claim you want to make. Estimation is the math that produces the number. Run the math without the argument and the number means nothing.
Most LLM summarization pipelines skip that step. A transcript arrives and the model starts producing facts immediately. It never asks whether the evidence is there, because your JSON schema asked for a value and a value is what it returns. That is where confident summaries of meetings where nothing happened come from, and the failure hides well, since the prose reads so smoothly.
If you are building complex integrations, my guide on keeping AI logic deterministic covers the same class of pitfall.
A three-stage architecture for honest summaries
What works is a pipeline that enforces discipline: three stages with a monotonic-weakening constraint. Here is how I have been structuring them.
- Extraction: a conservative pass that pulls raw facts and is not allowed to infer anything.
- Synthesis: the creative pass, where facts turn into claims tagged Observed, Inferred or Recommended.
- Audit: the stage that cannot add or improve text. It may only delete or weaken claims.
Audit is the stage developers get wrong, because we build systems to be helpful and a helpful auditor is just a second source of hallucinations. Constrain it to weakening only, so Observed can drop to Inferred but nothing climbs the other way, and the final output can never claim more than the source supports.
Enforcing the constraint in PHP
On a WordPress backend you need those operations enforced in code. Here is a naive wrapper around an auditor’s logic that stops it from helping too much.
<?php
/**
* Class bbioon_Summarization_Auditor
* Ensures LLM outputs stay grounded in source facts.
*/
class bbioon_Summarization_Auditor {
private $allowed_ops = ['delete', 'weaken', 'placeholder'];
public function process_claim($claim, $operation) {
if (!in_array($operation, $this-$>allowed_ops)) {
// Block any attempt to 'improve' or 'add' context
return $claim;
}
switch ($operation) {
case 'weaken':
$claim['category'] = 'Inferred';
break;
case 'placeholder':
$claim['content'] = 'Insufficient evidence in transcript.';
break;
case 'delete':
return null;
}
return $claim;
}
}Empty sections are the system working
Clients complain when a summary comes back with empty sections and assume the model broke. If the meeting was thin, the summary should be thin. An LLM summarization pipeline that fills eight sections whatever you feed it has stopped summarizing and started filling in a template.
When the abstention rate rises with the thinness of the input, the architecture is doing its job. An empty section is more useful than a fluent invention, which means part of the work is getting users, and yourself, comfortable with an “Insufficient Evidence” placeholder.
If you are wrangling more AI data than that, here is how to build a persistent context layer without losing the plot.
If LLM summarization pipelines are eating your dev hours, I can take it off your plate. I have been wrestling with WordPress since the 4.x days.
Where to start
A better model or a longer prompt will not fix hallucination. It takes a methodological change: put back the step that observational analysis never dropped. Do not estimate what you have not identified first. Users may prefer the smooth version, but a summary you can defend is worth more than one that only reads well.
