A floor plan is exact about shape and silent about identity
ICIDO's saved map is a filtered mesh of primary surfaces. ARKit's per-face classification labels wall, floor, ceiling, table, seat, window and door, and the scan keeps those while dropping small unclassified clusters (ARCHITECTURE.md §4). That short list is the whole vocabulary. The map knows a box-shaped thing standing at object height is furniture. It does not know what the furniture is — a commit on automatic room naming records the limit plainly: ICIDO cannot see a fridge, so a kitchen and a small dining room are the same eight numbers (dea6f74).
That is a boundary rather than a defect, and the recent work has been about answering each question on the correct side of it.
What geometry answers, it answers exactly
Spoken surroundings arrived first: a description that reads out the room, the clear floor ahead marched on the occupancy grid, and the nearest tagged spots and furniture with distance and direction (b298105). Turn-by-turn directions followed, collapsing the A\* polyline into real legs and speaking each one as it is reached — "go forward five meters, then turn left" — announced when the action changes rather than as the distance counts down (22c062b).
Both are computed, never generated. No model touches guidance, and the same commit shows why that rule earns its keep: the first version had left and right inverted, because a plain atan2(x, z) gives a heading whose sign is backwards for this coordinate frame. It was caught by testing against hand-worked L-shaped routes in both directions. A model asked for that number would have returned a plausible one.
Under VoiceOver, the app speaks through VoiceOver rather than over it (b298105).
What only a camera can answer
Camera questions — "what do you see", "what colour is the door" — go to a vision model fed from ARFrame.capturedImage (fd4738e). There is no second capture session: ARKit already owns the camera and two would contend for it, so the model sees the same frame tracking does. It arrives in the sensor's orientation, so it takes a fixed quarter turn, or every description is of a sideways room.
Some questions belong to both. "What's in front of me" went to the map alone at first, and that was reconsidered: the map knows how much clear floor is ahead, but not what the thing is (d53d39b). Ambiguous questions now get both halves, and the ordering carries the design — the map answer is spoken immediately because it is instant and exact about distance, and the camera answer follows a few seconds later. Making the whole reply wait for the model would penalise every question of that shape.
The balance was later shifted further toward the camera, leaving map-only exactly what a camera cannot do: which room you are in, how much clear floor is ahead, and how far away something is (acfb8c4).
Reading is a third job, and neither model gets it
Reading text uses Apple's VNRecognizeTextRequest, and it runs first rather than as a fallback (069b605). It transcribes instead of describing, answers in tens of milliseconds rather than seconds, reports per-line confidence so "I can't make that out" is answerable instead of guessed, and needs no download. Text comes back verbatim, because a door code is exactly what a paraphrase ruins.
That held even when a better vision model became available. iOS 27's system model can call an OCR tool, and reading was still deliberately left where it was (1e1e046).
Speaking without being asked raises the bar
Watch mode is an eye button beside the microphone on the scan and navigation screens: turn it on and the app keeps checking the camera and speaks up when what you are watching for appears, instead of being asked every few metres (d540415). The commit is explicit that an unprompted false alarm is worse than a wrong answer to a question, because the user did not choose the moment and cannot see what triggered it.
So the yes-or-no is requested as structured output rather than parsed out of prose; the bundled model, which has no guided generation, is prefix-parsed and fails toward silence; a match the model cannot describe is dropped, since that is the shape of a hallucinated yes; and repeats are suppressed for 45 seconds, keyed on the wording reduced to its letters, because the model rarely phrases a repeat sighting identically. Checks are paced four seconds apart for heat, not responsiveness — this is a full inference on top of a live session. It never speaks while a conversation is open, and sightings during one are dropped rather than queued: by then the camera has moved.
A later change gave watching a second half: saying what to do about a sighting, not only what it is, with the action editable in Settings and identical across both model backends (348399c).
Where the work runs
Speech recognition is forced on-device, and the feature refuses to run at all where the device does not support on-device recognition, rather than falling back to a server (88ba475). Nothing is written to disk, and the microphone button is hidden entirely where the device cannot do it. Scene memory — the record that makes "where did I leave my keys" answerable — is off by default, keeps text and never images, and is excluded from iCloud backup (cb7f320).
On iOS 27, camera questions can go to the model built into the operating system instead of the bundled weights, which removes a 370 MB download (1e1e046). That path is additive: the deployment target stays iOS 16, LiDAR reaches back to the iPhone 12 Pro, and most devices in the field will use the bundled model.
ICIDO has no accounts and no sign-in, and scans stay on the iPhone. The work described here is in TestFlight builds. ICIDO is coming to the App Store, and scanning requires the LiDAR sensor — iPhone 12 Pro and later Pro models.