A client came to me a few months back with what they called a “total disaster.” They had paid a junior dev to bolt a GPT-powered search assistant onto their high-traffic WooCommerce store, and on paper it was going to be the perfect shopping concierge. What it actually did was invent product features and hand out 90% discount codes that never existed. Customers were livid and the support team was drowning.
My first instinct, and I am a little embarrassed to admit it, was to double down on prompt engineering. I wrote a much bigger system prompt and tightened the JSON schemas, hoping to jailbreak-proof the thing. It held for about a day, until a user got it to give a “helpful” answer about a competitor’s pricing. The LLM’s logic was never the problem. We had not designed for user trust in AI, so the feature behaved like an opaque oracle instead of something a shopper could check.
The four legs of calibrated trust
WordPress developers tend to treat trust as security or uptime. With an AI feature it is closer to psychology. Smashing Magazine describes it as a four-legged stool: ability, benevolence, integrity and predictability. Knock out one leg and the whole thing tips over.
Predictability is the leg people forget. If your assistant answers the same question two different ways inside five minutes, users bail, because there is no way to build a mental model of how it behaves. With this client we stopped chasing blind trust and went after calibrated trust instead, where the user can tell where the AI is strong and where it is likely to trip. Being upfront about the limits bought us more goodwill than hiding them ever did.
Adding a confidence threshold
The hallucination problem on that store came down to shipping whatever the model produced. So we put a confidence check in front of it, reading the logprobs from the API to see how sure the model was about its own tokens, and dropped any response that scored below the line. The fake coupons stopped.
/**
* bbioon_check_ai_confidence
* Simple logic to filter AI responses based on a dummy threshold.
*/
function bbioon_validate_ai_response( $response_data ) {
$threshold = 0.85; // We want 85% confidence
// In a real scenario, you'd parse logprobs from your OpenAI/Anthropic call
$confidence_score = $response_data['average_logprob_score'];
if ( $confidence_score < $threshold ) {
return [
'status' => 'error',
'message' => 'I\'m not quite sure about that. Let me connect you with a human.',
'fallback' => true
];
}
return [
'status' => 'success',
'content' => $response_data['text']
];
}We also added a “Show Sources” toggle. Any product the AI recommended had to link to the real WooCommerce product ID, which covers the integrity leg: rather than asking people to take our word for it, we gave them a way to check. Support tickets fell 40% in a week.
What this means for your build
The API call is the easy part of an AI feature. Most of the work sits between what the model outputs and what the user was expecting to get:
- Aim for calibrated trust rather than total trust, so users know where the limits are.
- Handle errors with some humility. An “I don’t know” beats a confident wrong answer.
- Show the reasoning behind a suggestion, using explainability (XAI), so people can follow it.
- Watch the correction rate. If users keep rewriting what the AI produces, the ability leg is cracked.
This gets messy quickly. If you would rather not spend your week debugging someone else’s integration, send my team a note. We have probably run into your version of it already.
The question worth asking before your next AI feature ships is whether a user can tell when it is guessing.
