Aico: AI That Can Interact With Real Devices
Aico started from a simple problem: most AI agents work well when they have access to APIs, browser structure, or specially prepared software environments, but real-world device interfaces are much messier. Smartphones and computers contain native apps, custom-rendered interfaces, changing layouts, dialogs, and workflows that cannot always be accessed through a DOM or an API.
Aico is our approach to bridging that gap.
Aico is an AI agent designed to understand and interact with real smartphone and computer interfaces. It works together with Aico Link, our open-source hardware platform, which provides a physical observation and control layer between the AI agent and a compatible device.
Instead of depending exclusively on application-specific APIs or accessibility trees, Aico Link can observe the rendered screen through a video capture path and interact with the device through standard USB HID input. Aico then uses multimodal models to interpret the current interface, decide what action to take, execute it, observe the result, and continue the task.
The goal is not simply to make an AI click buttons. We are building toward agents that can complete longer, multi-step workflows while maintaining context, reacting to unexpected interface states, and keeping the user involved when needed.
Key features
- Real-device interaction across compatible smartphones and computers
- Multimodal screen understanding using configurable AI model providers
- USB HID control for keyboard, pointer, touch, and supported device actions
- Visual verification after actions instead of assuming an interaction succeeded
- Skills that provide reusable instructions and workflows for agents
- Persistent memory and session handling for longer-running tasks
- Voice interaction, including speech input and output where configured
- Human-in-the-loop control for interruption, redirection, and supervised execution
- Developer tools and benchmarks for testing agents across real devices, simulated environments, perception, planning, and memory tasks
- Open-source hardware and software, making the system inspectable and extensible
Aico is still evolving, and a major part of the project is exploring what reliable device-use agents actually require beyond a successful demo: better perception, grounding, memory, verification, recovery, and clear boundaries between what the agent can observe and what it is allowed to do.
We are especially interested in contributions from developers working on AI agents, multimodal models, embedded systems, mobile automation, GUI understanding, evaluation, and human-computer interaction.
You can explore the project, test the current implementation, report compatibility issues, suggest improvements, or contribute directly through the Aico Link open-source repository and community.