Where Model-Driven Screen Interaction Actually Runs Out of Room
Browser agents like Comet (Perplexity) are a good case study. They're capable — genuinely strong at page-level Q&A, tab-context synthesis, and supported web actions — but their control surface is bounded to the browser session: DOM, tabs, browser-accessible documents. No amount of model capability changes that boundary; it's architectural, not a model-quality problem.
The observation/action loop, one layer down
Aiden approaches this from the device layer instead of the browser layer. The documented dev-board implementation captures a connected screen via HDMI, and sends input via USB HID — meaning the "interface" being modeled isn't DOM structure or accessibility labels, it's raw pixels, same as any other real-world vision task.
That has real implications for the perception problem:
- No DOM to fall back on for grounding — button detection, layout interpretation, and text extraction all have to work from the same visual signal a human would use
- State tracking matters as much as single-frame interpretation — a button correctly identified in one frame can still fail if a dialog appeared, a screen is mid-load, or an earlier action silently didn't take effect
- The interaction loop is deliberately conservative: observe → interpret → propose one bounded action → verify the resulting state → escalate to the user when ambiguous, rather than chaining actions on unverified assumptions
flowchart LR
A[User instruction] --> B{Current control surface}
B --> C[Browser session]
C --> D["Tabs, webpages, web apps"]
B --> G[Connected device interface]
G --> H["Browser, native app, desktop app, or system dialog"]
H --> I[Observe visible state]
I --> J[Propose next action]
J --> K[User confirms, redirects, or stops]
K --> L[Execute supported interaction]
L --> M[Review outcome]
Why this is a genuinely different category, not a Comet knockoff
Worth being precise here: this isn't "Aiden vs. Comet." Fellou is explicitly agentic ("self-driving browser"), Dia has strong cross-tab context modeling (though more suggestive than autonomous in practice), Opera Neon is doing something more experimental around creation workflows. All of these are still operating within a browser's DOM/tab model. Aiden's distinguishing architectural choice is dropping down to raw screen capture + HID input specifically so the target interface doesn't need to be a browser at all — a native app, a desktop tool, a system dialog, anything with a visible screen and an input path, is fair game architecturally, even though real support today is bounded to documented Android/iPhone workflows.
The part relevant to grounding/vision folks specifically
The bring-your-own-multimodal-model design means the perception quality is entirely a function of whatever model you point it at — this is deliberately not a fine-tuned, task-specific vision model. That's a real tradeoff: you get to swap in whatever's strongest for your use case, but you also inherit whatever grounding weaknesses that general-purpose model has on UI-specific tasks (small text, overlapping elements, near-identical icons, mid-animation frames).
Curious whether others here have compared general-purpose multimodal models against UI-specialized grounding models for this kind of task, and at what latency/accuracy tradeoff.
Firmware (public): github.com/AidenAI-IO/aiden-firmware
Discord for technical discussion: discord.com/invite/bcJavjcnYz
