The Most Important Feature We’re Building Into Our AI Agent Is the Ability to Say “I Don’t Know”
AI agents are usually demonstrated by showing what they can do.
Write code.
Search the web.
Analyze documents.
Create plans.
Call APIs.
Automate workflows.
But while building Zorgax, the AI coordination layer inside MyZubster, I’ve started thinking that one of the most important capabilities of an agent might be the opposite:
knowing when it does not have enough evidence to act.
The dangerous part isn't hallucination alone
Developers already know about hallucinations.
A model can generate an answer that sounds perfectly reasonable and is completely wrong.
But once an AI system becomes an agent, the problem changes.
A hallucinated paragraph is annoying.
A hallucinated decision connected to an API can become an action.
Consider this:
LLM
↓
Confident answer
↓
API call
↓
Database write
↓
External action
Now uncertainty isn't just a language problem.
It's an infrastructure problem.
We started adding another possible output
In some Zorgax workflows, we're experimenting with explicit uncertainty states.
Instead of forcing the system to always produce:
YES
or:
NO
we allow:
MORE EVIDENCE REQUIRED
And sometimes:
TIE / HUMAN CHOICE
This sounds almost trivial.
It isn't.
A surprising amount of automation architecture assumes that every process eventually produces a machine-decidable answer.
Reality doesn't work that way.
Sometimes the available evidence genuinely isn't enough.
Confidence is not evidence
Imagine asking an AI agent:
Is this product idea validated?
A language model can easily produce ten paragraphs explaining why the idea looks promising.
But what evidence exists?
How many people were interviewed?
What problems did they report?
Were the answers independent?
Did anyone actually test the product?
Did they simply say “sounds cool”?
Would they use it?
Would they pay?
These are completely different claims.
So our preferred architecture is becoming:
QUESTION
↓
AVAILABLE EVIDENCE
↓
QUALITY CHECK
↓
ANALYSIS
↓
┌─────────────────────────────┐
│ Sufficient evidence? │
└─────────────────────────────┘
↓ ↓
YES NO
↓ ↓
Recommendation MORE EVIDENCE
REQUIRED
The AI isn't rewarded for always having an answer.
Sometimes the correct output is uncertainty.
This changes how you design agents
Once you accept that principle, several things change.
The agent needs to distinguish between:
KNOWN
INFERRED
ASSUMED
UNKNOWN
Those states should not be interchangeable.
For example:
KNOWN:
A test returned HTTP 201.
INFERRED:
The endpoint probably created the resource correctly.
UNKNOWN:
Whether the user finds the feature useful.
UNPROVEN:
Whether anyone will pay for it.
An AI system can help with all four categories.
But it shouldn't silently collapse them into:
FACT
The same problem appears in software engineering
Suppose Zorgax sees:
CI: green
What does that prove?
It proves the tests passed.
It does not prove:
the architecture is correct
the product is secure
users want the feature
production will never fail
the business will succeed
Green CI is evidence for a specific claim.
Nothing more.
I think this is an important mental model for AI systems:
Every piece of evidence has a scope.
And it becomes even more important with IoT
MyZubster is also exploring systems involving sensors and physical-world evidence.
Now imagine:
Sensor
↓
Data
↓
AI
↓
Decision
A sensor reports something unusual.
Should the AI immediately act?
Maybe.
But first:
Is the sensor healthy?
Is the reading within its calibration range?
Is the timestamp valid?
Is the measurement recent?
Do other sensors agree?
Is the data complete?
Was there a communication failure?
A better pipeline might be:
PHYSICAL EVENT
↓
SENSOR DATA
↓
PROVENANCE
↓
QUALITY CHECK
↓
ZORGAX
↓
CONFIDENCE / EVIDENCE GATE
↓
HUMAN DECISION
↓
ACTION
↓
MEASUREMENT
The AI becomes part of the evidence pipeline.
Not the source of truth.
“I don't know” needs architecture
There is another problem.
Simply prompting an LLM:
Say you don't know when uncertain.
isn't enough.
The surrounding software needs to understand uncertainty too.
For example:
const result = await zorgax.evaluate(evidence);
switch (result.status) {
case "SUPPORTED":
prepareRecommendation(result);
break;
case "INSUFFICIENT_EVIDENCE":
requestMoreEvidence(result.missing);
break;
case "CONFLICTING_EVIDENCE":
escalateToHuman(result);
break;
default:
failClosed();
}
Now uncertainty becomes a system state.
That is much more interesting than treating it as conversational wording.
Fail closed instead of inventing certainty
Security engineering already uses this idea.
If authorization cannot be verified:
DO NOT GRANT ACCESS
If payment cannot be verified:
DO NOT ACTIVATE ENTITLEMENT
If identity cannot be verified:
DO NOT MARK VERIFIED
Why shouldn't AI reasoning use a similar principle?
If evidence cannot support the claim:
DO NOT INVENT CONFIDENCE
This doesn't mean doing nothing forever.
It means returning the missing information required to continue.
INSUFFICIENT_EVIDENCE
Missing:
- additional observations
- independent confirmation
- baseline measurement
- verified identity
Now uncertainty becomes actionable.
Human-in-the-loop becomes more precise
“Human-in-the-loop” is sometimes used as a vague safety phrase.
But the interesting question is:
When exactly does the human enter the loop?
Not every operation needs approval.
Reading a public document?
Probably not.
Calculating a metric?
Probably not.
Preparing a draft?
Probably not.
Sending money?
Yes.
Changing verified identity?
Yes.
Publishing a consequential claim?
Potentially.
Acting when evidence is contradictory?
Definitely.
So instead of:
AI → HUMAN → EVERYTHING
we are exploring:
LOW-RISK / REVERSIBLE
AI can prepare or execute within scope
HIGH-RISK / IRREVERSIBLE
Human approval required
INSUFFICIENT EVIDENCE
Stop and request evidence
That makes the boundary much easier to reason about.
Agents need epistemic permissions
We already think about permissions like:
READ
WRITE
DELETE
EXECUTE
ADMIN
Maybe AI systems also need something like epistemic permissions:
OBSERVE
INFER
RECOMMEND
CLAIM
ACT
An agent might be allowed to infer something internally without being allowed to publish it as fact.
It might be allowed to recommend an action without being allowed to execute it.
It might be allowed to prepare a database change without being allowed to commit it.
That separation is becoming increasingly important in how we're designing Zorgax.
The architecture we're moving toward
The larger loop currently looks like this:
KNOWLEDGE
↓
EVIDENCE
↓
REASONING
↓
UNCERTAINTY CHECK
↓
RECOMMENDATION
↓
HUMAN APPROVAL
↓
ACTION
↓
MEASUREMENT
↓
LEARNING
The uncertainty check might be one of the most important components in that diagram.
Because without it:
REASONING
↓
CONFIDENCE
↓
ACTION
can become dangerous very quickly.
The goal isn't an AI that always answers
The AI industry often measures capability by asking:
How many tasks can the model complete?
For agents interacting with real systems, I think another metric will become increasingly important:
How reliably does the system recognize when it should not proceed?
A useful agent should be capable.
But it should also understand boundaries.
It should distinguish evidence from inference.
It should expose uncertainty.
It should fail safely.
And sometimes it should produce the least impressive AI answer possible:
I DON'T KNOW YET.
MORE EVIDENCE REQUIRED.
That might actually be a feature.
I'm building MyZubster + Zorgax in public and experimenting with AI agents, open-source infrastructure, human approval, IoT and evidence-driven automation.
I'm curious how other developers are handling this:
Does your agent architecture have an explicit “insufficient evidence” state, or does the model always have to produce an answer?