Blog

How Sitora decides it does not know

Daniel BrookCo-founder and CTOAugust 18, 20268 min read

Ask any grounded-answers system what its hardest problem is and, if it is being honest, the answer is not answering. It is deciding when not to. This post walks through how a Sitora agent reaches 'I do not know', because we think the mechanism deserves to be public.

Signal one: retrieval confidence

Before any model sees a question, we rank the site's passages against it. Strong questions light up specific passages: 'how much is a cleaning' lands hard on a pricing page. Weak matches, where the best passage shares only a stray word with the question, tell us the site probably does not cover this.

Below a confidence threshold we do not ask the model to be creative. We go straight to the handoff. A model never gets the chance to improvise on a question the site cannot support.

Signal two: the source requirement

When a model does answer, it must return the sources it used, and those sources must be passages we actually handed it. This is checked in code, not requested in a prompt. A reply that cites nothing, or cites pages outside the retrieved set, is rejected and becomes a handoff.

This inverts the usual failure mode. In most systems a hallucination looks exactly like an answer. In ours, an unsupported answer is structurally impossible to deliver: it fails validation before a customer ever sees it.

The refusal is a conversion feature

Owners sometimes worry the handoff makes their agent look limited. The data from our own use says the opposite: the handoff moment, done honestly, is where visitors leave their email. An 'I do not know, let me get you a person' with a working follow-up converts better than a confident wrong answer ever could, because the visitor knows the difference.

We publish the whole pipeline at /how-the-agent-works. If you are evaluating any grounded agent, ours included, ask the vendor to show you their refusal path first.

Give your website a voice

Free plan, grounded answers with sources, honest handoff. No card.

Build my agent