Perception gives JARVIS eyes: on the screen, through OCR, and through a camera. Eyes are also an attack surface and a privacy risk. Text on screen can carry instructions, a camera can record things it should not, and a mis-read gesture can move the wrong window. JARVIS’s perception layer is built on one principle: perception is evidence, never authority.
Libraries
| Job | Library | Note |
|---|---|---|
| Screen capture | Windows GDI | |
| OCR | Windows OCR, with RapidOCR as fallback | |
| Camera | OpenCV | |
| Hand, pose and face landmarks | MediaPipe Tasks | |
| Object detection | RT-DETR | Chosen for its Apache-2.0 licence |
The perception modules cover the camera, landmarks, gestures, a controller, activation, the processing pipeline, targeting, the screen, OCR, addressee cues and general awareness.
Privacy by construction
The privacy rules are built into the data types, not left to policy text:
- The camera is off by default. It can be reached only inside a purpose-named session with a bounded lease of at most 300 seconds.
- Frames cannot be kept. A camera frame object has no file path, cannot be saved, and refuses to be pickled (serialised). Raw frames are ephemeral. Only derived landmarks and evidence are retained.
- Screenshots are not persisted.
- OCR text is quarantined as possible prompt injection. Words read from the screen are data about the screen, never instructions to JARVIS.
The frame rule has a stated cost. The project notes that it “intentionally makes later pixel-level forensic replay impossible”. If something goes wrong, there is no stored video to review, only the landmarks and receipts. That trade was chosen on purpose: a system that does not keep pixels has no stored pixels to leak.
Screen grounding
Screen capture, OCR and a targeting module map regions of the screen to UI Automation or browser elements. JARVIS uses this to understand what it is looking at, and every resulting action still goes through the governance chain and the identity rules on the computer operation page.
Gestures (Z11)
The Z11 work adds governed hand gestures. The pipeline is: frame, landmarks, gesture state machine, target resolver, all eight gates, effect coordinator, SemanticOperator, then a verified receipt or compensation.
The project names eight gates, and each one can stop the gesture:
- Presence
- Zone
- Engagement
- Confidence
- Hysteresis, so a flickering recognition does not trigger anything
- Consequence, decided by
PolicyEnginelike any other effect - Preview, so the intended result is shown first
- Cancellation, so the owner can call it off
The first five ask, in different ways, whether this is really a deliberate gesture by the owner. The last three tie the gesture to governance: policy decides, the owner sees a preview, and the owner can cancel. A gesture system that moves windows on a false positive is worse than none, so any single gate is enough to stop a gesture.
What gestures may do
Gestures are limited to window actions: minimise and maximise, moving a window to a previewed rectangle, and moving a window to another monitor. The gesture-effects feature is off by default.
Two rules keep gestures from becoming a back door:
- POINT “never becomes effect authority”. Pointing can select or indicate, but cannot by itself make something happen.
- An open palm “can only remove authority”. It can stop or cancel, never start.
The HUD shows the gesture pipeline’s stage, the tracked skeleton, how fresh the data is, and why a gesture was refused, so a refusal is visible rather than mysterious. See HUD.
Status
- Screen observation and OCR are tested. Their registry rows were demoted for reachability in an August audit.
- Governed gestures are implemented. A safe local live proof ran on 19 August 2026 on isolated ports. It deliberately opened no camera and moved no window, so it shows the software path, not physical behaviour.
- Gesture recognition was once recorded LIVE. It was withdrawn to IMPLEMENTED when Z11 rewrote the recogniser. An older live reading does not carry over to new code.
- Hand tracking (journey J10) is fixture-tested only.
- Pose context failed its probe.
What is not proven yet
- Physical acceptance on the owner’s hardware has not been done. Camera frame rate, calibration precision, follow latency and false-activation rate are all recorded as not probed on the owner’s hardware.
- No live gesture has moved a real window in the evidence cited here. The live proof opened no camera and moved no window.
- Pose context failed its probe.
- Screen grounding has not been re-proven live since its reachability demotion.
- No object-detection accuracy figure is published here.