A receipt looks like a simple piece of paper.
At first glance, it contains a few obvious things:
A merchant name.
A date.
Some products.
A few numbers.
A tax amount.
A total.
Then the paper goes into a pocket, a wallet, a drawer, a bag, or a trash bin.
But technically, a receipt is something much more interesting.
It is an unstructured data record describing a financial transaction.
And the world produces billions of these records.
The problem is that receipts were never really designed for machines.
They were designed for humans.
A human can look at:
TOTAL: K184.50
and immediately understand what it means.
A computer does not.
A computer sees pixels.
Maybe it sees:
MARTIN'S
SUPERMARKET
19/09/26
Milk 24.50
Bread 18.00
Rice 75.00
Soap 32.00
VAT 35.00
TOTAL 184.50
But another receipt might look like this:
SHOPRITE
LUANSHYA
TXN: 8291938
19-09-2026 18:42
2 x Bread K18.00
1 x Milk K24.50
SUBTOTAL K60.50
VAT K9.08
CASH K100
CHANGE K30.42
And another might be:
AMZN.COM
Order #114-...
Sep 19
Wireless Mouse
$19.99
Shipping
$4.99
Tax
$2.10
Total
$27.08
Same concept.
Completely different structure.
This creates an interesting engineering problem:
How do we build a system that can look at almost any receipt in the world and turn it into reliable structured financial data?
That is what I call a Universal Receipt Intelligence System.
Not simply OCR.
Not simply receipt scanning.
Not simply an expense tracker.
The objective is much bigger.
We want to build a system that can understand receipts.
The Problem With Receipt Data
Traditional software expects structured input.
An API might receive:
{
"merchant": "Example Store",
"date": "2026-09-19",
"total": 184.50,
"currency": "ZMW"
}
That is easy.
The database knows what each field means.
But real-world receipts are not APIs.
They are documents.
Sometimes they are photographs.
Sometimes they are screenshots.
Sometimes they are PDFs.
Sometimes they are thermal-print receipts.
Sometimes they are handwritten.
Sometimes they are damaged.
Sometimes they are folded.
Sometimes they are extremely long.
Sometimes they contain multiple transactions.
Sometimes the receipt is printed in a language the system has never seen before.
Sometimes the merchant uses strange abbreviations.
Sometimes prices are aligned using spaces.
Sometimes they are aligned using dots.
Sometimes there are no labels at all.
And sometimes the total appears several times.
For example:
TOTAL
SUBTOTAL
VAT
TOTAL
TENDERED
CHANGE
A naive parser can easily mistake the wrong number for the purchase total.
This is why receipt intelligence is fundamentally a document understanding problem.
The Architecture
I would design the system as a pipeline.
┌───────────────────┐
│ Receipt Image/PDF │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Image Preprocessor│
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Document Detector │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ OCR / Text Engine │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Layout Analyzer │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Semantic Parser │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Entity Resolver │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Validation Engine │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Confidence Engine │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Structured Receipt│
└───────────────────┘
Each stage solves a different problem.
The important part is that we do not ask one model to magically understand everything.
We decompose the problem.
Step One: Receipt Ingestion
The system starts with an input layer.
It should accept:
JPEG
PNG
WEBP
PDF
HEIC
Screenshots
Camera captures
Email attachments
Scanned documents
An API might look like:
POST /v1/receipts
Content-Type: multipart/form-data
The response initially should not claim that the receipt has been completely understood.
Instead:
{
"receipt_id": "rcpt_8fd91",
"status": "processing"
}
The actual processing can happen asynchronously.
That matters because OCR and AI inference can be expensive.
A production system might use:
API
│
▼
Object Storage
│
▼
Message Queue
│
├── OCR Worker
├── Image Worker
├── Parser Worker
└── Validation Worker
This allows the system to scale horizontally.
Step Two: Image Intelligence
OCR quality depends heavily on image quality.
A photograph taken at an angle is different from a scanned receipt.
A thermal receipt photographed under poor lighting is different again.
Therefore, before OCR, we need preprocessing.
Typical operations include:
Resize
Deskew
Rotate
Crop
Denoise
Contrast normalization
Perspective correction
Shadow removal
Blur detection
Background removal
Suppose the camera captures:
/
/
/ RECEIPT
/
Perspective correction should transform it into something closer to:
┌──────────────────┐
│ RECEIPT │
│ │
│ Product 20 │
│ Product 40 │
│ │
│ TOTAL 60 │
└──────────────────┘
This seems like a small detail.
It is not.
Better input quality can dramatically improve everything downstream.
Step Three: OCR Is Only the Beginning
After preprocessing, we perform OCR.
But ordinary OCR gives us text.
We need more than text.
We need text plus geometry.
For example:
{
"text": "TOTAL",
"x": 420,
"y": 920,
"width": 90,
"height": 30
}
And:
{
"text": "184.50",
"x": 650,
"y": 920,
"width": 100,
"height": 30
}
The relationship between those two objects is valuable.
They appear on the same line.
They are horizontally aligned.
They are close together.
Therefore:
TOTAL → 184.50
is a strong relationship.
This is why receipt intelligence should preserve the original spatial information.
Step Four: Layout Understanding
Receipts have visual structure.
Even when the text changes, the structure often remains recognizable.
A receipt might contain:
HEADER
MERCHANT
ADDRESS
TRANSACTION METADATA
ITEMS
SUBTOTAL
TAX
DISCOUNT
TOTAL
PAYMENT
FOOTER
We can model this as a document tree:
Receipt
├── Merchant
├── Address
├── Transaction
│ ├── Date
│ ├── Time
│ └── Transaction ID
├── Items
│ ├── Item
│ ├── Item
│ └── Item
├── Financial Summary
│ ├── Subtotal
│ ├── Tax
│ ├── Discount
│ └── Total
└── Payment
├── Method
├── Tendered
└── Change
Now the problem becomes easier.
We are not simply extracting strings.
We are reconstructing a document.
Step Five: Merchant Detection
One of the first important entities is the merchant.
The system might detect:
SHOPRITE
But that does not necessarily mean the canonical merchant entity is exactly:
SHOPRITE
We may want:
{
"merchant": {
"display_name": "Shoprite",
"normalized_name": "shoprite",
"confidence": 0.99
}
}
Over time, merchant normalization becomes powerful.
For example:
SHOPRITE #123
SHOPRITE HOLDINGS
SHOPRITE LUANSHYA
SHOPRITE KITWE
can potentially be associated with a common merchant organization while preserving the individual branch.
This allows higher-level analytics.
Step Six: Currency Intelligence
Currency is surprisingly difficult.
A receipt might say:
K184.50
Another:
ZMW 184.50
Another:
184.50
The system needs to determine whether:
184.50
means:
USD
ZMW
EUR
GBP
ZAR
KES
NGN
The answer may come from several signals:
Currency symbol
Currency code
Merchant country
Merchant address
Known merchant location
Receipt language
Tax format
Historical transactions
User locale
The system should not blindly guess.
Instead, it should produce:
{
"currency": "ZMW",
"confidence": 0.97
}
or:
{
"currency": null,
"confidence": 0.41,
"requires_review": true
}
Uncertainty is better than fabricated certainty.
This is where the system becomes genuinely useful.
We want to transform:
2 MILK FULL CREAM 48.00
BREAD WHITE 18.00
SOAP BAR 12.50
into:
{
"items": [
{
"description": "MILK FULL CREAM",
"quantity": 2,
"total": 48.00
},
{
"description": "BREAD WHITE",
"quantity": 1,
"total": 18.00
},
{
"description": "SOAP BAR",
"quantity": 1,
"total": 12.50
}
]
}
But receipts can contain much more complicated formats.
For example:
MILK
2 @ 24.00 48.00
or:
MILK FULL CREAM 2X24.00
48.00
or:
001234 MILK 24.00
The parser needs to understand multiple representations.
Quantity and Unit Reasoning
A sophisticated receipt system should distinguish:
1 × 20.00
2 × 20.00
0.75 kg × 80.00
The third example is particularly interesting.
It may represent:
0.75 kg × K80/kg = K60
So the structured record should support:
{
"description": "Tomatoes",
"quantity": 0.75,
"unit": "kg",
"unit_price": 80,
"total": 60
}
This opens the door to much deeper financial intelligence.
The receipt is no longer just an expense.
It becomes a representation of what was purchased.
Step Eight: Financial Reconciliation
Now we introduce one of the most important components:
the validation engine.
Suppose the receipt contains:
Milk 24.50
Bread 18.00
Rice 75.00
Subtotal 117.50
VAT 17.63
Total 135.13
The system can calculate:
24.50 + 18.00 + 75.00 = 117.50
Then:
117.50 + 17.63 = 135.13
Everything reconciles.
That increases confidence.
But imagine OCR produces:
Rice 750.00
The total might no longer reconcile.
This creates a signal:
EXTRACTION ERROR POSSIBLE
The validation engine can then inspect nearby OCR candidates.
Maybe the image actually says:
75.00
This is where deterministic software and machine learning become extremely powerful together.
AI can interpret.
Rules can verify.
Step Nine: Multiple Candidate Extraction
One of the strongest ideas in this architecture is to avoid forcing the system to make an immediate decision.
Suppose OCR sees:
184.50
but image quality is poor.
The system can maintain candidates:
{
"field": "total",
"candidates": [
{
"value": 184.50,
"confidence": 0.91
},
{
"value": 184.80,
"confidence": 0.42
}
]
}
Then downstream validation can choose.
For example:
Candidate A
184.50
Candidate B
184.80
If:
subtotal + tax = 184.50
Candidate A becomes significantly stronger.
This architecture is much more robust than:
OCR → one answer → done
Step Ten: Semantic Understanding
A universal system should understand what a transaction represents.
Consider:
CANONICAL KEYBOARD
versus:
MILK
The system can classify products into categories:
Electronics
Groceries
Transport
Utilities
Office Supplies
Healthcare
Entertainment
Clothing
Software
Travel
Then:
{
"description": "Wireless Keyboard",
"category": "electronics",
"subcategory": "computer_accessories"
}
This enables analytics that traditional receipt scanners cannot provide.
Step Eleven: Merchant-Specific Learning
Universal does not mean treating every receipt identically.
The system should learn merchant-specific patterns.
Suppose thousands of receipts from the same supermarket are processed.
We may discover:
SKU
DESCRIPTION
QTY
PRICE
VAT
TOTAL
always appears in roughly the same layout.
The system can store a merchant profile:
{
"merchant": "example_store",
"layout_version": 4,
"known_fields": [
"sku",
"description",
"quantity",
"price",
"tax"
]
}
Future receipts can then be processed faster and more accurately.
The system evolves from:
generic understanding
toward:
generic understanding + merchant-specific knowledge
Step Twelve: The Receipt Knowledge Graph
Now things become much more interesting.
Instead of storing receipts as isolated JSON documents, we can create relationships.
Customer
│
├── bought
│
▼
Product
│
├── sold_by
▼
Merchant
│
├── located_at
▼
Location
Another relationship:
Receipt
├── contains → Product
├── issued_by → Merchant
├── paid_with → Payment Method
├── occurred_at → Date
└── belongs_to → User
Over time, this becomes a transaction knowledge graph.
The system can answer questions such as:
How much did I spend on groceries last month?
or:
Which merchants do I use most frequently?
or:
How much have I spent on computer equipment this year?
The receipt scanner has evolved into a financial intelligence system.
Step Thirteen: Duplicate Detection
Receipts can be uploaded multiple times.
Maybe the user scans the same receipt twice.
Maybe an email attachment is uploaded after a camera scan.
We need duplicate detection.
A simple approach can use:
Merchant
Date
Time
Total
Transaction ID
But images can also have perceptual hashes.
For example:
image_hash
combined with:
merchant + date + total
can produce a duplicate probability.
The system might return:
{
"duplicate_probability": 0.94
}
instead of silently creating another expense.
Step Fourteen: Fraud and Anomaly Detection
This is another powerful layer.
Imagine a user normally receives receipts around:
K20
K50
K100
K150
Then suddenly:
K48,000
appears.
That does not automatically mean fraud.
But it deserves attention.
The system can flag:
Unusual transaction amount
Other signals could include:
Duplicate receipt
Impossible date
Future transaction date
Invalid tax calculation
Repeated transaction ID
Altered image
Inconsistent totals
Suspicious OCR mismatch
Receipt intelligence can therefore become part of financial security.
Step Fifteen: Detecting Altered Receipts
This is one of the more advanced problems.
A receipt can potentially be edited.
For example:
TOTAL K120
could be manipulated into:
TOTAL K1,200
A serious system should not simply trust the image.
It can inspect:
font consistency
pixel patterns
spacing
compression artifacts
baseline alignment
character geometry
metadata
document structure
None of these signals alone proves manipulation.
But together they can create an anomaly score.
{
"integrity": {
"score": 0.91,
"flags": []
}
}
or:
{
"integrity": {
"score": 0.38,
"flags": [
"inconsistent_text_region",
"financial_reconciliation_failure"
]
}
}
The goal is not to declare guilt.
The goal is to identify receipts that require review.
Step Sixteen: Confidence Is a First-Class Data Type
A major design principle for this system should be:
Every important extraction should have confidence.
Not just:
{
"total": 184.50
}
but:
{
"total": {
"value": 184.50,
"confidence": 0.98
}
}
Similarly:
{
"merchant": {
"value": "Example Store",
"confidence": 0.96
}
}
and:
{
"currency": {
"value": "ZMW",
"confidence": 0.87
}
}
This makes the system explainable.
A downstream application can decide:
confidence >= 0.95
→ automatically accept
0.75–0.95
→ accept but flag
< 0.75
→ request review
The thresholds can be configured per use case.
Step Seventeen: Human-in-the-Loop Processing
There will always be difficult receipts.
Instead of pretending otherwise, build a review system.
Imagine the system says:
We are 62% confident that the total is K184.50.
The user sees the receipt and confirms:
Yes.
That feedback becomes valuable training data.
The architecture becomes:
Receipt
↓
AI Extraction
↓
Validation
↓
Confidence
↓
┌───────────────┐
│ High confidence│ → Automatic
└───────────────┘
┌───────────────┐
│ Low confidence │ → Human Review
└───────────────┘
This allows the system to improve continuously.
Step Eighteen: Event-Driven Architecture
At scale, I would not process everything synchronously.
A possible architecture:
API Gateway
│
▼
Receipt Service
│
▼
Object Store
│
▼
Queue
│
┌─────────────┼──────────────┐
▼ ▼ ▼
OCR Worker Image Worker Metadata Worker
│ │ │
└─────────────┼──────────────┘
▼
Parse Queue
│
▼
AI Parser
│
▼
Validation Engine
│
▼
Confidence Engine
│
▼
Receipt DB
This gives us independent scaling.
If OCR becomes the bottleneck, scale OCR workers.
If AI parsing becomes expensive, scale parser workers.
If validation is cheap, keep fewer validation workers.
The system becomes elastic.
Step Nineteen: Storage Design
I would separate the raw document from the structured interpretation.
Object storage:
receipts/
2026/
09/
rcpt_8fd91/
original.jpg
normalized.png
ocr.json
Database:
receipts
receipt_items
merchants
transactions
payments
receipt_events
receipt_reviews
The raw image should remain immutable.
The interpretation can evolve.
This distinction matters.
Suppose version 1 of the parser extracts:
Total = K180
and version 2 determines:
Total = K184.50
We should be able to reconstruct why.
Therefore, maintain processing versions:
{
"parser_version": "2.4.1",
"ocr_version": "7.2",
"processed_at": "2026-09-19T20:12:00Z"
}
Now the receipt has provenance.
Step Twenty: The Universal Schema
A possible universal receipt model could look like:
{
"id": "rcpt_8fd91",
"merchant": {
"name": "Example Store",
"confidence": 0.98
},
"transaction": {
"date": "2026-09-19",
"time": "18:42:11",
"transaction_id": "8291938"
},
"currency": "ZMW",
"items": [
{
"description": "Milk",
"quantity": 2,
"unit_price": 24.50,
"total": 49.00
}
],
"financials": {
"subtotal": 100.00,
"tax": 15.00,
"discount": 5.00,
"total": 110.00
},
"payment": {
"method": "cash",
"tendered": 120.00,
"change": 10.00
},
"confidence": 0.96
}
The schema should remain extensible.
Because universal systems fail when they assume the world is standardized.
The world is not standardized.
Step Twenty-One: APIs
The platform could expose several APIs.
Upload:
POST /v1/receipts
Retrieve:
GET /v1/receipts/{id}
Search:
GET /v1/receipts?merchant=shoprite
Analytics:
GET /v1/spending
Reprocess:
POST /v1/receipts/{id}/reprocess
Review:
POST /v1/receipts/{id}/review
Webhook:
receipt.processed
receipt.review_required
receipt.failed
Now other applications can build on top of the intelligence engine.
Accounting software.
Expense management.
Retail applications.
Personal finance applications.
Tax software.
Business intelligence systems.
Procurement platforms.
Insurance platforms.
And even AI agents.
Step Twenty-Two: Making the System Agent-Friendly
This is where receipt intelligence becomes particularly interesting.
Imagine an AI agent receiving:
"Analyze my business expenses."
Instead of searching through images manually, it queries:
GET /v1/transactions
and receives normalized data.
The agent can then reason over:
merchant
category
amount
date
product
tax
payment method
The receipt system becomes an observation layer for financial agents.
The AI does not need to understand pixels.
It receives structured reality.
Step Twenty-Three: Learning Without Destroying Reliability
Machine learning systems have a dangerous tendency.
They can become increasingly confident while becoming increasingly wrong.
Therefore, I would keep deterministic rules around critical financial fields.
For example:
subtotal + tax - discount ≈ total
must be checked mathematically.
Quantity × unit price should approximately equal item total.
Change should approximately equal:
tendered - total
Tax should fall within plausible constraints.
Dates should be valid.
Currency codes should be valid.
Transaction IDs should be consistent.
AI proposes.
Rules verify.
That division of responsibility is extremely important.
Step Twenty-Four: Measuring the System
Accuracy should not simply be:
"Does OCR work?"
We need field-level metrics.
For example:
Merchant Accuracy
Date Accuracy
Currency Accuracy
Total Accuracy
Tax Accuracy
Item Accuracy
Quantity Accuracy
Category Accuracy
We can define:
Exact Match Accuracy
Field-Level Precision
Field-Level Recall
Numeric Error Rate
Reconciliation Rate
Human Review Rate
Duplicate Detection Accuracy
One particularly useful metric is:
Straight-Through Processing Rate
Meaning:
What percentage of receipts can be processed correctly without human intervention?
That is a much more meaningful production metric than OCR character accuracy alone.
Step Twenty-Five: The Feedback Loop
Every correction becomes data.
Suppose users repeatedly correct:
"TOTAL" → "SUBTOTAL"
The system learns.
Suppose a particular merchant's receipt format changes.
The system detects a drop in confidence.
That can trigger:
merchant template drift detected
Now the platform can adapt.
This creates:
Receipts
↓
Extraction
↓
Validation
↓
Human Corrections
↓
Training Data
↓
Improved Models
↓
Better Extraction
The system becomes progressively better at understanding the world.
The Deeper Idea
The most interesting part of this project is not receipt scanning.
It is the transformation:
Physical document
↓
Pixels
↓
Text
↓
Structure
↓
Entities
↓
Relationships
↓
Financial meaning
↓
Machine-readable knowledge
That is a general pattern for intelligent software.
We are taking something designed for human perception and converting it into something machines can reason about.
The same architecture can eventually apply to:
Invoices
Bills
Purchase orders
Delivery notes
Bank statements
Warranty documents
Tax documents
Contracts
Shipping documents
Receipts are simply an excellent starting point.
Building the MVP
I would not begin by trying to support every receipt in the world.
Start with:
1. Image upload
2. OCR
3. Merchant extraction
4. Date extraction
5. Currency detection
6. Line-item extraction
7. Total extraction
8. Mathematical validation
9. Confidence scores
10. JSON API
Then test it against hundreds or thousands of real receipts.
Not perfectly clean demo receipts.
Real receipts.
Wrinkled receipts.
Dark photographs.
Partially cut receipts.
Different countries.
Different currencies.
Different languages.
Different merchants.
Different printers.
Different layouts.
Then expand.
A Possible Technology Stack
A practical implementation could use:
API:
FastAPI / Django / Node.js
Workers:
Python / Rust
Queue:
Redis Streams / RabbitMQ / Kafka
OCR:
Tesseract / PaddleOCR / cloud OCR
ML:
PyTorch
Database:
PostgreSQL
Object Storage:
S3-compatible storage
Search:
OpenSearch
Cache:
Redis
Observability:
OpenTelemetry
Rust would be particularly interesting for high-throughput document pipelines.
Python would be excellent around machine learning.
PostgreSQL would handle the structured financial data.
Object storage would preserve original documents.
The architecture does not require every component to be written in the same language.
Use the right tool for the right layer.
The Final Architecture
At maturity, the system could look like this:
┌─────────────────────┐
│ Camera / Upload │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Document Processing │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ OCR + Vision Engine │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Layout Understanding│
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Semantic Extraction │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Entity Resolution │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Validation Engine │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Confidence Engine │
└──────────┬──────────┘
│
┌───────┴────────┐
▼ ▼
High Confidence Low Confidence
│ │
▼ ▼
Auto Process Human Review
│ │
└───────┬────────┘
▼
┌─────────────────────┐
│ Receipt Knowledge │
│ Layer │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ APIs / Analytics / │
│ AI Agents / Apps │
└─────────────────────┘
This is no longer a receipt scanner.
It is an intelligence layer between the physical financial world and software.
Conclusion
A receipt is deceptively simple.
It is a tiny financial document that encodes merchant identity, products, quantities, prices, taxes, discounts, payment information, time, location, and transaction information.
Yet most software still treats it as an image.
That is the opportunity.
The Universal Receipt Intelligence System changes the abstraction.
Instead of asking:
How do I read this receipt?
we ask:
How do I understand this transaction?
That difference completely changes the architecture.
OCR becomes only one component.
Computer vision becomes another.
Natural-language understanding becomes another.
Entity resolution becomes another.
Mathematical validation becomes another.
Knowledge graphs become another.
And confidence becomes the glue holding the system together.
The final product is not simply:
image → text
It is:
image
↓
document
↓
transaction
↓
structured knowledge
↓
financial intelligence
That is the real engineering challenge.
And it is also the interesting part.
Because once machines can reliably understand receipts, they can begin understanding a much larger portion of the physical economy.
Every supermarket receipt.
Every restaurant bill.
Every hardware-store purchase.
Every fuel receipt.
Every business expense.
Every transaction that was once trapped inside a piece of paper can become structured data.
And when enough of that data becomes machine-readable, the receipt stops being the end of a transaction.
It becomes the beginning of computation.
Code is not only about building software that creates new things. Sometimes it is about teaching software how to understand the things humans have already created.
