I Built My Own Personal AI Assistant — Meet IRIS
I've always wanted to build my own personal assistant.
Not just another chatbot that answers questions, but something that could actually live on my computer and help me get things done.
So I started building IRIS.
What started as an idea slowly turned into a real desktop AI assistant running locally on my Mac.
IRIS can now chat with me, use voice, work with files, search the web, remember information, understand screenshots, control applications, plan tasks, and interact with my computer.
And honestly, getting it to this point wasn't exactly straightforward.
There were plenty of bugs, broken implementations, slow models, permission issues, packaging problems, and a lot of moments where something simply refused to work. 😂
But that's also what made building it interesting.
What exactly is IRIS?
IRIS is my attempt at building a personal AI assistant, rather than just another conversational AI application.
The idea is simple:
Instead of me adapting to a bunch of different tools, I want one assistant that can work with those tools for me.
I can talk to IRIS normally, and depending on the request, it can decide what it needs to do.
For example, a request might involve:
User
↓
Understand request
↓
Determine what needs to happen
↓
Select the appropriate tool
↓
Check permissions
↓
Execute the action
↓
Observe the result
↓
Respond to the user
That's the direction I wanted IRIS to move toward from the beginning.
What can IRIS do?
💬 Natural conversations
At the core of IRIS is a local language model running through Ollama.
I'm currently using Qwen 2.5 3B for the conversational side.
This gives IRIS its ability to understand requests, answer questions, reason about tasks, and interact naturally.
The important part for me was keeping the core assistant local, rather than building everything around a cloud API.
🎙️ Voice interaction
I didn't want IRIS to feel like an application where I always have to type.
It can accept voice input through the Mac's microphone and convert speech into text.
The flow is:
My voice
↓
Microphone
↓
Speech recognition
↓
IRIS
↓
Response
IRIS can also speak its responses back using the system's voice output.
There's even an audio toggle in the interface, so I can choose whether IRIS responds verbally.
📁 Working with files
IRIS can interact with files on the computer.
For example, it can:
- Search for files
- Read files
- Create files
- Edit files
For actions that can modify the filesystem, IRIS has a permission/confirmation layer rather than blindly executing everything.
That's important because giving an AI access to your computer without any safeguards is obviously not something I wanted to do.
🌐 Web search and browsing
IRIS can also search the web and retrieve information.
The system can:
Search
↓
Find relevant results
↓
Retrieve useful content
↓
Process the information
↓
Generate an answer
This allows IRIS to go beyond the information available inside its local model.
🖥️ Application control
One of the things I really wanted was the ability to tell my assistant to interact with applications.
IRIS can launch supported applications on my Mac.
For example:
"Open Calculator."
And IRIS can actually open it.
This might sound simple, but it's one of the steps toward making an AI assistant something that can do things, rather than simply tell you how to do them.
🧠 Memory
Another major part of IRIS is memory.
I didn't want every conversation to feel like starting from zero.
IRIS has different layers for things such as:
- Working memory
- Long-term memory
- Personal profile
- Conversation state
- Memory retrieval
- Memory updates
- Memory deletion
The idea is that information can be stored and later retrieved when it's relevant.
For example:
"Remember that this project is called IRIS."
Later:
"What was the project I told you about?"
The assistant should be able to connect those two conversations.
👁️ Vision
IRIS can also understand screenshots.
For this, I integrated a local vision model:
Granite 3.2 Vision 2B
The workflow is:
Screenshot
↓
Image processing
↓
Vision model
↓
IRIS understands the screen
↓
Response
This opens up a completely different direction for the project.
Instead of only understanding text, IRIS can start understanding what is actually happening on the screen.
🤖 Planning and autonomous tasks
Another major part of IRIS is task planning.
For more complicated requests, the assistant can break a goal into smaller steps.
For example:
Goal
↓
Task 1
↓
Task 2
↓
Task 3
↓
Observe results
↓
Recover from failures
↓
Complete task
I built components for:
- Planning
- Task decomposition
- Multi-step execution
- Observation
- Recovery
- Task state
- Goal tracking
- Agent memory
- Evaluation
The goal is to eventually move from:
"Tell me how to do this."
to:
"Do this for me."
🔐 Permissions and safety
This was something I didn't want to ignore.
An AI assistant with access to your computer needs boundaries.
IRIS has a permission system that distinguishes between different types of actions.
Some actions can happen directly, while potentially risky actions can require confirmation.
For example, filesystem and terminal operations can require permission before execution.
The idea is simple:
IRIS should be capable, but it shouldn't blindly do everything it can.
🧩 Plugins
I also added a plugin system so IRIS doesn't have to remain limited to the tools I originally built.
Plugins can provide additional capabilities while following a common interface.
That means the architecture can eventually grow from:
IRIS
├── Files
├── Web
├── Applications
├── System
└── Memory
into something more extensible:
IRIS
├── Core tools
├── Plugins
├── External capabilities
└── Future integrations
🏗️ How IRIS is structured
The project has grown into multiple layers rather than being one giant Python file.
At a high level:
IRIS
│
┌───────┴───────┐
│ │
Interface Brain
│ │
┌─────┴─────┐ ┌────┴─────┐
│ │ │ │
Chat Voice Memory Planning
│
Tool Layer
│
┌─────────────┼─────────────┐
│ │ │
Files Web System
│ │ │
Applications Search Computer
│
Vision
The exact implementation is more detailed, but this is the basic idea.
The interface talks to the assistant engine, the engine can use memory and planning, and the tool layer gives IRIS the ability to interact with the computer and the outside world.
🛠️ Tech stack
Some of the main technologies behind IRIS are:
- Python
- PySide6 — desktop interface
- Ollama — local model runtime
- Qwen 2.5 3B — local language model
- Granite 3.2 Vision 2B — vision
- Nomic Embed — embeddings
- PyTorch
- PyInstaller — macOS application packaging
The project currently runs as a proper macOS application, so I don't need to open VS Code and run a Python command every time I want to use it.
I can simply launch IRIS like a normal application.
😅 The difficult part wasn't always the AI
One thing I learned while building IRIS is that making an AI assistant isn't just about connecting an LLM to a chat box.
A lot of the work was actually around the systems surrounding the model.
Things like:
Voice input
Getting the microphone, speech detection, silence detection and packaged application permissions working correctly.
Memory
Making sure memories can actually be stored, retrieved, updated and forgotten.
Tool execution
Making sure the assistant can select and execute the correct tool without breaking everything around it.
Vision
Getting screenshot analysis fast enough to actually be useful.
Permissions
Allowing IRIS to interact with the computer without giving it unlimited freedom.
Packaging
Turning a Python project with all these dependencies into an actual Mac application was its own challenge.
Those problems taught me something important:
Building an AI product is much more than building the model.
The model is only one part of the system.
🚀 What's next?
I'm not considering IRIS finished.
There are still a lot of things I want to improve.
The next direction is focused less on simply adding more features and more on making IRIS actually smarter and more useful.
Some of the things I want to work on:
- Better tool selection
- Better understanding of natural language
- Stronger conversational context
- More reliable memory
- Better autonomous task execution
- Improved computer control
- Better screen understanding
- More robust error recovery
- Faster responses
- More integrations
Eventually, I want to be able to give IRIS a goal and have it figure out the steps itself while keeping me in control of important actions.
Final thoughts
IRIS started because I've always wanted to build my own personal assistant.
It could have stayed as an idea.
Instead, I decided to actually build it.
There were plenty of times when things broke, features had to be rebuilt, and the project became much more complicated than I initially expected.
But that's probably my favorite part of the project.
I'm not just experimenting with an AI model anymore.
I'm building a system around it.
Something that can see, listen, remember, reason, use tools and interact with my computer.
It's still far from the assistant I imagine in my head.
But now I can actually open my Mac, launch IRIS, talk to it and watch that idea come to life.
And honestly, I'm just getting started. 🚀
