We spend a lot of time asking how intelligent an AI model is.
How accurate is it?
How fast is inference?
How well does it perform against a benchmark?
How many parameters does it have?
Those questions matter. But when I build AI systems intended to operate outside a research notebook, there is another question I consider even more important:
What happens when the model is wrong?
Because eventually, it will be.
A model can achieve impressive accuracy and still fail on an unfamiliar input. It can produce a high-confidence prediction that is incorrect. It can perform perfectly in controlled testing and behave differently once environmental conditions change.
This is why I do not design AI systems where the model has unrestricted authority over the final action.
I prefer a different architecture:
AI proposes. Rules validate. Humans intervene where necessary. Everything important is logged.
I have applied variations of this principle across robotics, computer vision, language systems, and quantitative finance.
The more I work with AI, the more convinced I become that reliable AI engineering is not about trusting models more.
It is about designing systems that know when not to trust them.
A Model Prediction Is Not a Decision
Machine-learning models are probabilistic systems.
That is what makes them powerful.
They can recognise patterns across inputs that would be impractical to describe using thousands of manually written rules.
But probabilistic output should not automatically become operational authority.
Consider an object detector.
A typical detection might contain:
class: pedestrian
confidence: 0.94
bounding_box: [x1, y1, x2, y2]
It is tempting to interpret 0.94 as:
"There is definitely a pedestrian here."
That is not what it means.
The model is expressing confidence according to what it learned from its training process.
It does not know that a camera has shifted.
It does not understand that lighting conditions have suddenly changed.
It does not know that another sensor disagrees with it unless we explicitly build that information into the wider system.
That distinction matters enormously.
In my computer-vision and robotics work using YOLOv11, ByteTrack and StrongSORT, I treat detections as inputs to a decision pipeline, not final decisions themselves.
The model generates evidence.
The system decides what can safely be done with that evidence.
The Architecture I Prefer for High-Stakes AI
A simplified version looks like this:
Raw Input
|
v
+---------------+
| AI / ML Model |
+---------------+
|
v
Confidence Checks
|
v
Deterministic Rules
|
+------+------+
| |
v v
Approved Uncertain
| |
| v
| Human Review
| |
+------+------+
|
v
Logging / Audit
|
v
Action / Output
Each part has a different responsibility.
AI models
Excellent for:
- pattern recognition
- classification
- prediction
- perception
- generalisation
- processing complex or unstructured data
Deterministic logic
Excellent for:
- safety limits
- policy enforcement
- numerical constraints
- permissions
- validation
- kill switches
- behaviours that must remain predictable
Humans
Excellent for:
- ambiguity
- contextual judgment
- unusual edge cases
- high-impact approvals
- decisions where consequences matter more than speed
Logging and observability
Essential for answering:
- What did the model predict?
- Why did the system accept it?
- Which rule was triggered?
- What data was available?
- What action followed?
- What happened afterwards?
This separation of responsibilities creates something much stronger than simply connecting a powerful model to an API and allowing it to act.
Robotics Makes the Problem Easy to See
Imagine a mobile robot navigating a warehouse.
Its camera detects a clear path.
Confidence: 96%.
Should the robot accelerate?
Not necessarily.
That prediction may have come from one frame.
An obstacle could be partially occluded.
The camera could have experienced motion blur.
The robot itself might be unstable.
A tracking system may have only observed the scene for a fraction of a second.
So instead of using:
if detection.confidence > 0.90:
move_robot()
I prefer logic closer to:
def can_navigate(detection, tracker, platform):
if detection.confidence < CONFIDENCE_THRESHOLD:
return False, "confidence_too_low"
if tracker.track_age < MIN_TRACK_AGE:
return False, "track_not_stable"
if platform.linear_acceleration > MAX_JITTER:
return False, "platform_unstable"
if not tracker.consistent_across_frames:
return False, "temporal_validation_failed"
return True, "navigation_authorised"
None of these individual checks are especially sophisticated.
That is the point.
They are understandable.
Testable.
Auditable.
And deterministic.
The neural network handles what neural networks are good at: perception.
The surrounding software handles what traditional software engineering is good at: constraints.
A high-confidence prediction is therefore not permission to act.
It is one piece of evidence.
I Applied the Same Philosophy to AI-Assisted Trading
Financial systems make the same problem less physically visible but potentially much more expensive.
When building Sentinel Quant, I wanted AI to contribute to portfolio analysis and decision support without giving an AI model unrestricted control over capital.
You can explore the live project here:
Sentinel Quant: AI Trading Terminal
Sentinel Quant is a modular quantitative portfolio-management system built around an important principle:
The intelligence layer and the authority layer should not be the same thing.
The AI components can analyse information such as:
- market conditions
- momentum signals
- regime changes
- portfolio opportunities
- confidence levels
- short-, medium- and longer-term allocation signals
But before an action can be accepted, it must survive deterministic controls.
A simplified example looks like this:
def validate_trade(trade, portfolio):
if portfolio.drawdown > MAX_DRAWDOWN:
activate_kill_switch()
return False
if trade.position_size > allowed_position_size(portfolio):
return False
if portfolio.beta_exposure > MAX_BETA:
return False
if trade.confidence < MIN_MODEL_CONFIDENCE:
return False
if trade.symbol in RESTRICTED_ASSETS:
return False
return True
The model does not get to negotiate with these rules.
If maximum drawdown has been exceeded, the answer is no.
If a proposed position violates portfolio constraints, the answer is no.
If an exposure limit has already been reached, the answer is no.
This separation is important.
The AI can say:
"Based on the patterns I see, this trade looks attractive."
The risk engine can answer:
"That may be true, but this action violates the system's risk policy."
The risk engine wins.
Every time.
Why I Think "Zero Trust" Also Makes Sense for AI
Zero-trust architecture is usually discussed in cybersecurity.
The basic idea is straightforward:
Never automatically trust. Always verify.
I think the same mindset is useful when designing autonomous or semi-autonomous AI systems.
A model output should not be trusted simply because it came from the model.
Instead:
Model proposes
|
v
System verifies
|
v
Rules constrain
|
v
Human reviews when necessary
|
v
Action executes
This becomes increasingly important as AI systems gain access to tools.
An AI that only generates text has one type of risk.
An AI that can:
- execute trades
- control robots
- modify databases
- send emails
- deploy software
- approve transactions
- modify infrastructure
- communicate with customers
has an entirely different risk profile.
The question changes from:
"Was the response accurate?"
to:
"What is this system actually authorised to do?"
That is an engineering problem, not simply a machine-learning problem.
Multi-Model Systems Multiply Uncertainty
Another project that reinforced this lesson for me is Signlytic, a bidirectional British Sign Language translation system.
Its pipeline combines multiple AI components, including computer vision, language processing and speech synthesis.
Conceptually:
Video
|
v
Sign Recognition
|
v
BSL Representation / Gloss
|
v
Language Model
|
v
Natural-Language Output
|
v
Speech Synthesis
The interesting engineering problem is not simply whether each model performs well individually.
It is what happens when uncertainty moves through the pipeline.
Imagine:
Recognition confidence = 0.68
|
v
uncertain BSL token
|
v
LLM interprets token
|
v
fluent English sentence
The final sentence could sound extremely convincing even though the first stage was uncertain.
This is a common danger in multi-model systems.
Fluency can hide uncertainty.
A downstream model can turn an uncertain upstream prediction into an output that appears authoritative.
So uncertainty needs to travel through the architecture too.
For example:
if sign_prediction.confidence < LOW_CONFIDENCE:
request_repeat()
elif sign_prediction.confidence < REVIEW_CONFIDENCE:
show_uncertainty_indicator()
else:
continue_translation()
The exact fallback depends on the application.
It could:
- ask the user to repeat the input
- display an uncertainty warning
- provide multiple possible interpretations
- fall back to text
- route the result for human verification
- refuse to execute a high-impact action
What matters is that failure behaviour is deliberately designed.
Deterministic Logic Is Not Obsolete
There is sometimes an assumption that increasingly capable AI will eliminate the need for traditional rules.
I think the opposite is happening.
The more capable AI becomes, the more important deterministic controls become around it.
Consider a simple rule:
if transfer_amount > authorised_limit:
reject_transaction()
Could we ask an LLM whether the transaction should be allowed?
Of course.
But why would we?
For an absolute financial limit, deterministic code provides something a language model cannot:
a guarantee.
The same applies to many constraints.
if temperature > SAFE_LIMIT:
emergency_shutdown()
if user.role != "admin":
deny_operation()
if portfolio.drawdown > MAX_DRAWDOWN:
stop_trading()
if robot_distance_to_human < SAFETY_RADIUS:
emergency_stop()
AI is useful precisely where simple rules stop being sufficient.
It does not mean we should replace rules that already solve a problem perfectly.
Human-in-the-Loop Is an Architecture, Not a Failure
Another assumption I regularly see is:
"Human review is temporary. Once AI becomes good enough, we can remove humans."
That may be appropriate for some applications.
But it should not be the default objective.
Sometimes human involvement is the correct permanent design.
Suppose an automated system processes 100,000 cases.
Perhaps:
- 94,000 are sufficiently clear for automatic processing
- 5,500 require additional deterministic checks
- 450 have meaningful uncertainty
- 50 could produce serious consequences if handled incorrectly
Removing humans entirely may save time.
But a better architecture might automate the 99,550 cases where machines are strongest and route the remaining 450 to people.
That is not inefficient AI.
That is intelligent allocation of responsibility.
The objective should not always be:
Maximum automation.
A better objective is often:
Maximum useful automation within acceptable risk.
Design the Failure Path Before the Success Path
When starting an AI product, teams naturally focus on the ideal flow.
Input -> Model -> Correct Prediction -> Action
I think teams should spend just as much time designing this:
Input
|
v
Model
|
+---- Wrong prediction
|
+---- Low confidence
|
+---- Conflicting signals
|
+---- Missing data
|
+---- Tool unavailable
|
+---- Policy violation
|
+---- Unexpected state
For every branch, ask:
What should happen now?
That question forces the team to define system behaviour before production defines it for them.
A robust AI architecture should be able to answer questions such as:
- What confidence level is required before an action can occur?
- Which actions can AI execute autonomously?
- Which actions require human approval?
- What happens when models disagree?
- Which constraints can never be overridden?
- What information is logged?
- Can an action be reversed?
- Where is the kill switch?
- What happens if an external API disappears?
- How does the system communicate uncertainty to its users?
These questions may sound less exciting than comparing foundation models.
They are also the questions that determine whether a system survives contact with the real world.
Observability Is Part of AI Safety
There is another layer that developers sometimes add too late:
auditability.
For important actions, I want to know more than what happened.
I want to reconstruct why it happened.
A useful event record might contain:
{
"timestamp": "2026-08-30T09:42:12Z",
"model": "signal-model-v4",
"prediction": "BUY",
"confidence": 0.87,
"risk_check": "PASSED",
"position_limit": "PASSED",
"drawdown_check": "PASSED",
"human_approval": true,
"final_action": "EXECUTED"
}
If something goes wrong three weeks later, this information becomes invaluable.
Without observability, teams end up asking:
"Why did the AI do that?"
With proper instrumentation, the question becomes:
"Which model output, system state, threshold and rule combination produced this action?"
That second question is much easier to investigate.
Reliability Comes From the System, Not Just the Model
Suppose two teams use exactly the same foundation model.
Team A
User -> Model -> Action
Team B
User
|
v
Input Validation
|
v
Model
|
v
Confidence Evaluation
|
v
Policy Engine
|
v
Risk Controls
|
v
Human Escalation
|
v
Action
|
v
Audit Log
Both teams have access to identical AI capabilities.
But they do not have equally reliable products.
The competitive advantage increasingly lies outside the model itself.
It lies in:
- orchestration
- validation
- evaluation
- guardrails
- observability
- tool permissions
- fallback behaviour
- risk management
- human escalation
- infrastructure
In other words:
The model may provide the intelligence, but engineering provides the reliability.
The Question I Ask Before Deploying AI
Whenever I work on a system where AI influences real-world actions, I return to one question:
What happens when this model is confidently wrong?
If the answer is:
"The system performs the action anyway."
I am uncomfortable with the architecture.
If the answer is:
"Another layer validates the decision, constraints are enforced, uncertain cases are escalated, and everything is traceable."
Now we have something much closer to production engineering.
AI systems do not need to be perfect.
They need to be designed with the assumption that they are not perfect.
That difference sounds subtle.
It changes the entire architecture.
Final Thoughts
I do not think the future of AI engineering is simply bigger models making increasingly large numbers of autonomous decisions.
I think it is about increasingly capable models operating inside increasingly well-designed systems.
Systems that understand:
- where probabilistic reasoning is useful
- where deterministic rules are mandatory
- where human judgment matters
- where actions need approval
- where uncertainty should stop execution
- and where every important decision needs to be traceable
That philosophy has influenced how I approach robotics, computer vision, accessibility technology and quantitative finance.
In Sentinel Quant, for example, the AI contributes intelligence to the decision-making process, but deterministic risk controls and human governance define the boundaries within which that intelligence can operate.
And that is ultimately the distinction I think matters most:
A powerful AI model can make a recommendation. A well-engineered system decides whether that recommendation deserves to become an action.
The future is not just smarter AI.
It is AI with boundaries.
#ai #machinelearning #softwareengineering #python #artificialintelligence