The open source, self-hosted platform for building, orchestrating and governing AI agents.
Obsidian AI is an agent orchestration platform for organisations that need AI agents to do real work, under real controls, on infrastructure they own.
It provides a single control plane for the full agent lifecycle: designing agents on any model, connecting them to tools and knowledge, coordinating them as teams and automated workflows, deploying them to the channels people use, and measuring, improving and governing them in production. Every capability shares one data model, one execution engine and one security boundary.
This article introduces the platform, its architecture and the capabilities it delivers today.
Obsidian AI is a full-stack application with three components:
- a Next.js 16 and React 19 frontend with the chat playground, agent builder, visual workflow editor, evaluation and observability screens, vaults and administration;
- a FastAPI backend of 29 routers and 173 REST endpoints that runs agents, teams, workflows, schedules and background jobs;
- a Node.js sidecar that connects agents to WhatsApp.
Data lives in SQLite by default, with no setup required, or in MongoDB by changing a single environment variable. Both are first-class: 32 data models are implemented for each, and the core features behave identically on either.
Next.js frontend
playground · agent builder · workflow canvas · evals
observability · vaults · channels · admin
|
/api proxy
|
FastAPI backend
+-------+-------+-------+-------+-------+-------+-------+
| | | | | | | |
LLM MCP RAG & DAG Sched- Eval & Docker Async
layer client memory engine uler optim. sandbox jobs
|
OpenAI · Anthropic · Gemini · Ollama · OpenRouter · any OpenAI-compatible
|
SQLite (default) / MongoDB
|
WhatsApp bridge (Node.js, Baileys)
The platform covers seven areas, each explored below:
| Area | What it provides |
|---|
| Agents and tools | Model-agnostic agents with built-in, custom, MCP and sandbox tools, and agents that can write their own tools |
| The playground | Streaming chat with artifacts, live previews, file analysis and rich, structured output |
| Orchestration | Multi-agent teams and visual DAG workflows with retries, approvals and cron schedules |
| Knowledge | Retrieval-augmented generation, self-maintaining long-term memory and Markdown vaults |
| Quality | Evaluation suites, a prompt auto-optimizer, version history with diffs, traces and cost analytics |
| Channels | WhatsApp, including groups, voice notes and cloned-voice replies |
| Governance | Per-user encryption, granular permissions, approvals, two-factor authentication and isolated execution |
One builder, every model
An agent in Obsidian AI references a provider rather than a specific SDK, so the same agent can move between models without being rebuilt. Six provider types are supported: OpenAI, Anthropic, Google Gemini, Ollama for fully local models, OpenRouter, and any OpenAI-compatible endpoint. Each provider can be tested from the UI, lists its available models, and can be exported and imported individually or in bulk. Current-generation Claude models use adaptive reasoning effort.
The agent builder brings an agent's entire configuration into one place:
- provider, model and system prompt, with the option to load from or save to the prompt vault;
- tools, MCP servers and knowledge bases;
- Markdown notes and Claude skills to inject into its instructions;
- which tools require human approval;
- whether it may create its own tools;
- its long-term memory, version history, optimizer runs and Docker sandbox.
Agents export to a portable JSON file. On import, tools, MCP servers and knowledge bases are matched by name in the destination workspace, and anything that cannot be matched is reported as a warning rather than failing silently.
| Source | What it covers |
|---|
| Built-in | Web search and URL fetching, available to every agent |
| Custom | Python functions or REST API calls, created from templates (Web Search, Weather, Calculator, Date and Time, API Request, Generative UI) or from scratch with a JSON Schema |
| MCP | Any Model Context Protocol server over stdio, SSE or Streamable HTTP, with a connection test before saving |
| Sandbox | Nine tools for shell, files, glob, grep, Python and Node.js inside a Docker container |
When permitted, an agent can propose a new tool mid-conversation with create_tool, or fix one it built earlier with edit_tool. If the agent supplies a description but no implementation, the platform generates the Python handler itself. The proposal appears as a review card in the chat, and nothing is saved until a person approves it. Agents can only edit tools that a model created, never tools a person wrote.
Work that outlives the conversation
Some MCP tools start jobs that take minutes, such as rendering a design or generating a report. Instead of promising to "check back later", an agent can call schedule_async_check. The backend then re-invokes the status tool on an interval, uses the model to classify the result as complete, failed or still pending, and posts the outcome into the conversation once it lands. A global badge tracks every job in flight.
A computer of its own
Agents and teams can be given a dedicated Docker container: Ubuntu 24.04 with Python, Node.js, Git, build tools and ripgrep, limited to 512 MB of memory and one CPU. Port 8080 inside the container is published to the host, so an agent can build and serve a working web application, then surface the files it wrote as artifacts in the chat.
Part Three: A Playground Built for Agents
The playground is where agents, teams and workflows are used day to day. It is designed to show what an agent is doing, not just what it says.
Artifacts with patch-based editing
Substantial output, such as a web page, a component, a document or a dataset, is produced as an artifact in a side panel rather than buried in the chat. Twelve artifact types are supported, including HTML, JSX and TSX, SVG, Python, TypeScript, Markdown, JSON and LaTeX. Web types render as live previews, and every artifact can be expanded, copied or downloaded.
Editing is where the design pays off. When a user asks for a change, the agent does not regenerate the whole file. It responds with targeted search-and-replace patches that are applied to the current version. A one-line change to a long page costs one line of output, not the whole page again. The platform also detects code that a model forgot to wrap and recovers it into an artifact automatically.
Output that shows its working
The stream carries more than text. Depending on what the agent is doing, the interface renders:
- execution plans, with steps that tick off as they complete;
- terminal output for shell and command tools;
- file trees for directory listings;
- source citations collected from search and fetch calls;
- progress indicators for multi-round tool use;
- generative UI cards for structured results such as weather, statistics, calculations and search results;
- reasoning from models that expose it, and a full execution trace for every reply.
Files and images
Images attached to a message are passed to vision-capable models. Documents (PDF, Word, Markdown and plain text) are extracted and indexed for that conversation automatically, so a user can attach a report and immediately ask questions about it.
Long conversations that keep going
When a conversation reaches 80% of the model's context window, older messages are summarised into a compact record while the ten most recent stay verbatim. The session continues without the user noticing a limit.
Every response can be rated with a thumbs up or down, and starter suggestions help new users discover what an agent can do.
Part Four: Teams and Workflows
Single agents are useful. Most real processes, however, involve several steps, several areas of expertise, and at least one point where a person needs to sign off.
Multi-agent teams
Agents can be grouped into teams that operate in one of three modes:
| Mode | Behaviour |
|---|
| Coordinate | A router model selects the most suitable agent for each message |
| Route | All agents respond in parallel, and a synthesiser merges the best answer |
| Collaborate | Agents run in sequence, each building on the previous agent's output |
In every mode, agents can also delegate directly. The call_teammate tool lets an agent ask a named colleague a question mid-turn and use the answer in its own response. Delegation is deliberately capped at one hop, which bounds latency and cost and makes delegation loops impossible. Each agent also sees a short, read-only summary of what its teammates remember about the user, and a team can share a single sandbox container.
Visual DAG workflows
For processes that should run the same way every time, workflows are drawn on a visual canvas as directed acyclic graphs. Templates such as Linear Pipeline, Parallel Agents and Conditional Routing provide a starting point.
| Node | Purpose |
|---|
| Start / End | Entry point and one or more terminal outputs |
| Agent | Runs an agent with a task, drawing on earlier nodes' outputs |
| Condition | A model chooses a branch from a routing prompt; skipped branches propagate through the graph |
| Approval | Pauses for a human decision, with a mandatory timeout that either fails or auto-approves |
| Map | Runs an agent over every item in a list, with a concurrency limit, returning a JSON list or concatenated text |
Workflows are built to survive the real world:
- Concurrency. Independent branches run in parallel; fan-in waits for every dependency.
- Retries. Each node can retry with fixed or exponential backoff, a maximum delay, and an optional list of retryable errors.
- Timeouts. Nodes can be given a time limit, and approvals can never wait forever.
- Safe data passing. Outputs are referenced with
{{ nodes.<id>.output.<path> }}. This is plain string substitution over a constrained grammar, with no expression evaluator, so a workflow cannot be used to inject code.
- Live runs. Runs stream node by node, can be cancelled, and are recorded with per-node status, output and trace for later review.
Schedules
Any workflow can run on one or more cron schedules, each with its own input, and schedules can be paused and resumed. Jobs are stored in the database, so they survive restarts, and missed runs are collapsed rather than replayed in a burst. One execution engine serves manual and scheduled runs on both databases, so a workflow behaves identically however it is started.
Importing from n8n
Teams already invested in n8n can paste an exported workflow and convert it. Control flow maps across structurally: triggers become the start node (with any cron timing detected and offered as a schedule, never switched on automatically), IF, Switch and Filter nodes become conditions, loops become map nodes, Wait nodes become approvals, and no-op nodes are rewired out. Integration nodes, such as HTTP requests, Gmail or Slack actions, become agent steps whose task describes in plain language what the original node did.
The importer is explicit about its limits. Every converted node keeps its original n8n type and parameters, credentials are listed by name, imports are capped at 500 nodes, and any expression that cannot be translated exactly is left in place with a warning.
Human-in-the-loop, everywhere
Any tool on any agent can be marked as requiring approval. Execution pauses and the request appears in a global notification badge, from which it can be approved or rejected anywhere in the application. The same mechanism covers playground chats, team conversations, workflow runs and messages arriving through WhatsApp.
Part Five: Knowledge and Memory
An agent is only as useful as the context it can draw on. Obsidian AI separates that context into distinct layers, each suited to a different kind of information.
| Layer | What it holds | How the agent uses it |
|---|
| Knowledge bases | Uploaded PDF, Word, Markdown and text documents | Relevant passages retrieved for each question |
| Conversation files | Documents attached in a chat | Indexed for that conversation only |
| Long-term memory | Durable facts learned from conversations | Recalled across sessions, per user |
| Markdown vault | Obsidian-style notes in folders | Injected into the prompt, or copied into a knowledge base |
| Prompt vault | Reusable system prompts | Loaded into agents and updated by the optimizer |
| Skills vault | Named instruction bundles for Claude agents | Attached to an agent and added to its instructions |
Memory that maintains itself
After a conversation, a reflection step reads the exchange and distils at most five durable facts: preferences, project context, decisions made and corrections given. Pleasantries and one-off questions are discarded, and artifact content is stripped out first so it never leaks into future sessions. Each memory has a key, a category and a confidence score; a newer fact with the same key replaces the old one, and the most recent 50 are injected into future conversations.
Memory that people can see and change
Memory is not a black box. Every memory can be reviewed, edited or deleted in the agent builder. An agent's entire memory can be exported as a single Markdown file, corrected in any text editor and imported back, with existing keys updated in place and new ones created. Memory can be switched off per agent. What an agent believes about its users is always visible, and always under their control.
Documents are split into overlapping chunks and indexed locally, using LEANN on Linux and macOS and FAISS with a local sentence-transformer model on Windows. No external vector database or embedding API is required. Responses show which knowledge bases contributed, and flag any that have not yet been indexed.
Part Six: A Built-In Quality Loop
The hardest part of running agents is not building them. It is knowing whether a change has made them better or worse. Obsidian AI treats this as a first-class concern.
Evaluation suites
An eval suite is a set of test cases for an agent, each pairing an input with an expected result. Three graders are available:
| Grader | Passes when |
|---|
| Exact match | The response matches the expected output exactly |
| Contains | The response includes the expected text |
| LLM judge | A judge model scores the response from 0 to 1 against the intent of the expected output, with written reasoning |
Suites can be run against any agent at any time, and results are stored per suite and per agent, so a regression is visible the moment it appears.
Prompt auto-optimizer
The optimizer turns real usage into measured improvements:
1. Collect sample recent conversation traces for the agent
2. Analyse extract failure patterns, each with a frequency,
a severity and example conversations
3. Propose rewrite the system prompt so that every pattern maps
to a specific instruction or example, without
removing behaviour that already works
4. Validate run an eval suite against the current and proposed
prompts and record both scores
5. Review show the failure patterns, the rationale, a
side-by-side diff and the score change
Nothing changes until a person accepts. An accepted proposal becomes a new agent version, and it can also be saved to the prompt vault. The model used for analysis is configurable in settings.
Versions with diffs
Every change to an agent's configuration is snapshotted. The builder shows a field-by-field before-and-after diff of each version, and any retained version can be restored. Versions from the previous 72 hours are kept, with the latest always preserved, and because the current state is snapshotted before a rollback, a rollback can itself be undone.
Traces and analytics
Every model call and tool call in a chat or workflow run is recorded as a trace span with its timing, tokens and inputs. Any reply or run can be opened in a trace view. The observability dashboard aggregates those spans into token usage and cost over time, latency by model, call counts, error rates and latency per tool, and cost per agent, priced for current models from Anthropic, OpenAI, Google and xAI, including Anthropic's prompt-cache reads and writes.
Together, these form a closed loop: observe how an agent behaves, propose an improvement, prove it against known cases, approve it, and roll back if needed.
Part Seven: Agents Where People Already Are
The web playground is where agents are built and tested. For many use cases, though, the people an agent serves will never open a browser. Obsidian AI connects agents directly to WhatsApp.
Connecting a channel
A channel is created in the interface and linked by scanning a QR code from WhatsApp's Linked Devices screen, with connection status streamed live to the page. There is no business account, webhook setup or per-message fee. Each channel is assigned an agent, which can be changed at any time without re-linking, and every linked channel reconnects automatically when the server restarts.
Each message runs through the agent's full pipeline: tools, retrieval, memory and approvals. Every contact gets a persistent session, so the agent remembers the conversation.
Built for how people really message
Messaging is not chat in a web page, and the bridge is designed around that:
- Bursts of messages. People send several short messages in a row. On MongoDB deployments, messages are buffered for 10 to 20 seconds (configurable) and answered together. If more arrive while the agent is working, the run is cancelled and restarted with the full context, so the reply never answers half a thought.
- Threaded replies. When a buffered burst contains several separate questions, a lightweight pre-pass groups them and decides which message to quote, so replies land against the right question.
- Natural presence. The agent shows as "typing" while it works.
- Groups. Agents can take part in group chats, where each participant keeps their own session and replies quote the message they answer.
- Context. Image captions and quoted replies are passed to the agent along with the text.
- Access control. A channel can answer everyone, or only whitelisted contacts and groups, with an optional message for anyone else.
- Approvals on the move. If an agent needs approval for a tool, the request appears in the web interface and the conversation resumes once it is actioned.
Voice in and out
Voice notes are handled end to end on local hardware:
incoming voice note
-> transcribed locally with faster-whisper (CPU)
-> agent pipeline: tools, RAG, memory, approvals
-> reply spoken by the best available engine
Qwen3-TTS (GPU: 9 preset voices, or a cloned voice)
Pocket TTS (CPU: 8 voices)
Kokoro (CPU, last resort)
-> encoded as an Opus voice note and sent
Voice replies are enabled per channel, can be limited to specific contacts, and the engine can be pinned or left on automatic. With Qwen3-TTS, a user can record a sample in the browser by reading a guided script, or upload one, and the agent replies in their own voice. The reference audio is processed once and cached. No audio is sent to an external service.
The bridge is built on Baileys, a widely used open source WhatsApp Web library. Because it is not an official WhatsApp API, it is best suited to personal and internal use, and operators should review WhatsApp's terms for their deployment.
Part Eight: Security and Governance
An agent platform holds API keys, executes code and acts on behalf of its users. Security is part of the architecture, not an afterthought.
| Layer | Protection |
|---|
| Authentication | NextAuth on the frontend and signed JWTs on the backend, with optional TOTP two-factor authentication |
| Transport | Login, registration and other sensitive payloads are AES-encrypted in the browser before they are sent |
| Secrets at rest | Provider keys, vault secrets and 2FA seeds are encrypted with a master Fernet key; the encryption layer also supports per-user AES-256-GCM envelope keys |
| Secrets in use | Provider keys are never returned to the browser; providers can reference a vault secret instead of holding a key; the secrets vault only ever displays masked values |
| Permissions | Admin and user roles, plus per-user permissions to create agents, teams, workflows, tools and knowledge bases, and to manage providers and MCP servers. Changes take effect immediately, without waiting for a token to expire |
| Integrations | Separate API client credentials (X-API-Key) for machine-to-machine access, created and revoked per user |
| Abuse protection | Rate limits of 60 requests a minute per user and 100 per API client by default |
Executing untrusted code
Agents that write and run code need somewhere safe to do it, so there are two levels of isolation.
User-defined Python tools run in a separate, short-lived interpreter for every call. It starts in isolated mode with only the standard library, receives an environment stripped of secrets, works in a temporary directory deleted afterwards, and is held to a 30-second timeout, CPU limits, a 512 MB memory cap and a 1 MB output limit. A faulty or malicious tool cannot read the server's keys or bring the server down.
The Docker sandbox is for anything that should never touch the host at all, with each agent or team confined to its own resource-limited container.
Combined with approval gates on sensitive tools, reviewed tool proposals and reviewed prompt changes, operators keep fine-grained control over what agents are permitted to do, and when.
Getting Started
Obsidian AI requires Python 3.12 or later with uv, and Node.js 18 or later. Docker is needed only for the sandbox, and a GPU only for the highest-quality voice engine.
git clone https://github.com/sup3rus3r/obsidian-ai.git
cd obsidian-ai
cd backend && uv sync && cd ..
cd frontend && npm install && cd ..
cd wa-bridge && npm install && cd ..
# configure backend/.env and frontend/.env.local (see the README)
npm run dev
A single npm run dev from the project root starts the frontend, the backend and the WhatsApp bridge together. Open the frontend at http://localhost:3000, add a provider, and create your first agent.
Two optional extras: a free Tavily API key enables the built-in web search tool, and the sandbox image is built once with:
docker build -f backend/Dockerfile.base -t obsidian-webdev-base:latest backend/
Design Principles
A few principles have guided the project throughout.
- Self-hosted first. The platform, its data and, where the hardware allows, its models and voice engines run on infrastructure you control.
- Model-agnostic. No agent, team or workflow is tied to a single provider.
- Humans in control. Tool approvals, workflow approval nodes, reviewed tool proposals and reviewed prompt changes keep people in charge of consequential actions.
- Transparent state. Memories, prompts, traces and versions are visible, editable and exportable rather than hidden.
- Portable by default. Agents and providers export and import, memory exports to Markdown, and workflows can be brought in from n8n.
Where Things Stand
Obsidian AI is open source under the AGPL-3.0 license and under active development. The platform currently comprises roughly 62,000 lines of code across the frontend, backend and WhatsApp bridge, with 29 API routers, 173 endpoints and 32 data models implemented for both SQLite and MongoDB.
The foundations are in place. The focus now is on broadening documentation, extending test coverage and growing the range of integrations. Contributions, issues and feedback are all welcome, and the repository includes contribution guidelines and a security policy.
The aim is straightforward:
A single, open platform where AI agents can be built, connected, governed and improved, on infrastructure you own.
Get involved:
If the project is useful to you, a star on GitHub helps others discover it.