Visual Subsystem

Computer Use & Visual Grounding

Kyro enables autonomous desktop interaction using low-latency screen capture, coordinate transformation, and native input synthesis without relying on heavyweight accessibility frameworks.

Set-of-Marks (SoM) Visual Grounding

Instead of guessing arbitrary pixel coordinates, Kyro can apply a visual Set-of-Marks (SoM) tag overlay to screenshots. Each interactable target (buttons, text inputs, menus) receives a numbered bounding badge:

• The LLM reasons over high-level mark numbers (click_mark(14)).
• The Rust engine maps mark 14 to precise display coordinates (x, y).
• Multi-monitor scale factors and DPI differences are automatically normalized via overlay.rs.

Native Input Synthesis via Enigo

Low-level mouse and keyboard inputs are emitted through the native enigo crate. Operations include smooth cursor movement, left/right clicks, double clicks, text typing, and key combinations.

Safety Override:Users can immediately abort computer actions at any time by moving the physical mouse, pressing the global hotkey, or clicking 'Abort Computer Use' in the floating HUD.

Ghost Mode Virtual Display (:99)

On Linux systems, Kyro can run desktop automation tasks headlessly inside an isolated virtual X11 display (DISPLAY=:99 powered by Xvfb). This prevents the agent from hijacking your primary screen and mouse cursor while you work.