Operating someone’s real computer is where assistants usually fail quietly. They click the wrong window because it has the same title. They type into a field that has moved. They report success because the click went through, whether or not anything happened. JARVIS’s desktop layer is designed around those failures. Every action is previewed, gated by the governance chain, journaled, and verified by reading the world back.
The interaction ladder
To act on an application, JARVIS chooses the most structured interface available, in this order:
- Native API. Most precise, and states can be read back directly.
- UI Automation. The Windows accessibility tree, with named controls and states.
- Keyboard. Deterministic, but blind to layout.
- Synthetic mouse. The last resort.
Each rung down is more fragile, so JARVIS descends only when it has to. Coordinate-based clicking is withheld by default. When it is used, it is announced, because a silent coordinate click is the most likely way to act on the wrong thing. Directly changing a window through its raw native handle is refused in production, because Windows can recycle a handle for an unrelated window.
Window identity and the 20-second lease
Two windows can share a title and a position. So JARVIS identifies a window by its handle, process ID, window class and process-creation generation, and never by title or rectangle. Once a target is resolved, it gets a 20-second identity lease. If the lease expires before the action runs, the action is refused with “target identity is unknown or expired”. That refusal was recorded live in the capability model on 16 September 2026.
Why refuse rather than re-resolve silently? A re-resolution could pick a different window than the one the owner meant. Refusing is cheap. Acting on the wrong window may not be.
When a phrase such as “close the editor” matches more than one window, JARVIS asks which one instead of choosing.
Verification, rollback and undo
After every action, JARVIS re-reads the geometry and state it expected to change. The outcome is one of:
- VERIFIED: what was read matches what was expected.
- MISMATCH: the action is rolled back rather than reported as done.
- FAILED: the effect did not happen.
- ROLLED_BACK: the change was reversed.
Undo is exact. The window controller reverses “this exact journalled action, never whichever action happened last.” So if two things happened, undoing one cannot accidentally reverse the other.
JARVIS also yields to the owner. If the owner types or clicks during the final focus boundary before an effect, JARVIS stands down and never retries that effect. A person’s hands on the keyboard outrank a plan.
Browser
JARVIS drives the owner’s visible Chrome through Playwright. It finds elements semantically, by role and accessible name, rather than by pixel positions or fragile selectors. Downloads and raw click-and-type are refused in production. Nothing a web page says can reach policy, approvals or the E-stop. Page text is data, never an instruction.
The Advanced Systems branch, which is not merged into production, adds a DOM scene reader. It understands relationships across frames and shadow roots, which production cannot see into today. It also adds dom-target-check/1, which refuses a target that has moved, been replaced, been covered or changed since it was observed. Two rules keep browser observation from turning into authority: “CSS viewport rectangles are observation data, never native screen leases”, and “CURRENT/STALE is an observation comparison, never click authority.” Typing and form filling on this path are still pending validation.
Screen grounding
Screen capture, OCR and a targeting module map regions of the screen to UI Automation or DOM elements. Browser regions feed the same path without gaining any native authority. A generic “invoke” with no independent check that the application reached its goal stays UNVERIFIED. The system does not credit itself for a click whose result it cannot see. Text recognised on screen is quarantined as possible prompt injection. See perception.
Scaled displays
On the Advanced branch, UI Automation geometry was corrected for Windows display scaling. An owned, disposable window was moved across four displays, with each position read back in physical pixels. A later move was then refused by the E-stop. This ran on a disposable fixture window, not the owner’s applications.
What is live and what is tested
| Area | Rung | Evidence |
|---|---|---|
| Opening apps, window control, files, multi-monitor | LIVE-PROVEN | Capability model, 16 September 2026 (historical) |
| Browser with semantic targeting | LIVE-PROVEN | Capability model, 16 September 2026 (historical) |
| Stale-target refusal | LIVE-PROVEN | Capability model, 16 September 2026 (historical) |
| Bounded native application operation | LIVE-PROVEN | Re-recorded 19 September 2026 (historical) |
| Registry desktop rows | TESTED | Deliberately not live-probed |
| Screen grounding | TESTED | Tests |
| Frames, shadow roots, target checks | INTEGRATED | Advanced branch only |
| DPI-correct geometry | INTEGRATED | Advanced branch, fixture window |
What is not proven yet
- Live readings are historical. They were recorded on 16 and 19 September 2026.
- Full-universe installed-app journeys are still owed at release. Live evidence so far covers a bounded set of applications and actions.
- The registry does not live-probe desktop mutations. It skips them on purpose to avoid disturbing the owner’s screen, so their registry rows stay TESTED.
- Iframes and shadow DOM are absent from production. Browser typing and filling on the Advanced path are unvalidated.
- Screen grounding was demoted for reachability in an August audit and has not been re-proven live.