Omnistant
An agent that notices what you forgot — before you're gone.
It watches a camera feed, keeps an append-only log of what it has seen and where, and acts on that log without being asked. Built solo in seven days for the #AllThingsAgentic hackathon, on Google ADK and Gemini 3.5 Flash, live on Cloud Run.
The problem
People with ADHD don't forget that they own things. They forget what happened to them. The wallet isn't lost — the memory of setting it down is.
Every existing tool answers the wrong question. Reminder apps need you to remember to set a reminder. Trackers only work on the things you remembered to tag. Both require the exact faculty that isn't available at the moment you need it.
So the agent has to be the one paying attention. The user cannot be the trigger — that constraint is what forces every architectural decision downstream: continuous observation, scheduled autonomy, and a memory that is a log rather than a conversation.
The rule everything follows from
The model never sources a fact.
Gemini does the seeing and the wording. Every claim the agent makes is computed from rows in a table. This is what makes the failure mode survivable: an agent that improvises “your AirPods are probably in the kitchen” is worse than useless to someone with ADHD — it sends them to the wrong room with false confidence.
What it does
Leave detection
You say you're heading out. The agent takes a camera frame, asks Gemini what is actually in it, and compares that against the routine it has learned for that destination. Anything missing comes back with a last-known location — then it keeps watching rather than answering once, ticking items off as the camera sees them.
Item recall
Searches the observation log and returns every sighting, newest first, each with a confidence that decays with age. A sighting from ten minutes ago is an answer; one from yesterday is a lead, and the agent says which. Costs zero model calls — it is pure SQL.
Daily timeline
Reconstructs the day from the log and has Gemini narrate it. The model is handed the events and forbidden from inventing any; when the log is thin, the agent says the log is thin.
The product
Four choices the model had no part in.
Why it is an agent and not a chat box
It watches without being asked
The camera stays on and the agent tracks the two transitions a person would also notice: appeared, and left view. That second one is how “I watched you put your AirPods in your bag” gets recorded — not by seeing the bag, but by seeing the AirPods stop being on the desk.
It learns when you leave
The pre-departure brief fires in the 25 minutes before the time this routine has actually been observed to happen — the median of the largest cluster of your own departures, not an hour somebody guessed at setup.
It follows you out
Two minutes after you leave, if something is still missing, it posts to Slack with the location — so you can decide in one glance whether it is worth turning back.
It cancels its own reminders
Every pending nudge is re-checked against the log before delivery. If the item has been seen since the scan that raised it, nothing is sent. Telling someone they forgot the keys they are holding is exactly the noise that gets an assistant muted.
It writes its own packing list
Nobody configures a routine. A new destination starts from a sensible default; after that, any item carried on 60% or more of the last eight trips is promoted into that destination's routine automatically. The work check grows to include your badge because you keep taking your badge.
The honesty mechanism
A sighting is not a fact about the present. So confidence decays, and the decay drives the phrasing — which is the entire point.
confidence × method_weight × exp(-ln2 × age_hours / half_life)
Method weights rank how the agent came to know: visual 1.0, because the camera saw it · voice 0.85, because you said so · manual 0.8 · inferred 0.55, because we guessed. The half-life is six hours, so a camera sighting starts at 0.90 and is worth 0.06 a day later.
Architecture
Tools are the surface; workflows are the substance. A tool never decides what is missing — it calls a workflow that computes it. That boundary is the reason the agent cannot hallucinate a sighting: there is no code path where a model's opinion becomes a stored fact.
State is split by durability requirement rather than by data type. Postgres holds what must survive — observations, learned routines, leave scans, due-dated nudges, and the ADK conversation sessions themselves. Redis holds what is fine to lose: session state, camera frames, the last-seen index, all on a TTL. Sessions live in Postgres specifically so the service can run above one instance; on a platform that scales to zero, nothing can wait in memory.
Continuous watching is only affordable because most frames are never sent. Each tick is compared client-side against the last frame actually transmitted — a 32×24 greyscale luma fingerprint, mean per-pixel delta, thresholded. A still desk costs zero API calls; only real movement earns one.
When it breaks
Degrade to something honest, never to something confident.
Which produced the sharpest lesson of the build: graceful degradation is correct, and it means a broken deploy looks healthy. Five bugs in seven days failed silently for exactly that reason — I wrote all five up in Your Fallbacks Are Hiding Your Bugs.
Four bugs worth the build
Durable sessions that shipped broken, behind a green deploy
ADK's in-memory session service keeps a conversation in one process's RAM. On Cloud Run — which scales to zero and out to four — that is not a session store, it is a way of losing conversations. So I wired up the Postgres-backed one: new dependencies, tested locally, deployed green. requirements.txt, which is what the Dockerfile actually installs, was never regenerated. Production silently fell back to in-memory. The fix was undone by packaging, and nothing looked broken.
There is now a packaging test, and /health names the session backend it settled on rather than claiming to be up.
A migration that was healthy and 404ing at the same time
I moved the deployed service off the AI Studio key onto Vertex AI. Env vars set, revision deployed, container healthy, /health reporting ok — and every model call returning 500. The location defaulted to us-central1, matching the region everything else deployed to. Vertex does not serve current Gemini models from regional endpoints: gemini-3.5-flash 404s there and 200s on global. What makes it quiet is that an older model is regionally available, so a deployment pinned to 2.5 works and upgrading the model breaks it with no code change to explain why.
Fixed by defaulting to global, with the probe table left in the comment so the next person to tidy it back to a region has to read the evidence first. The first probe reported 404 everywhere — a zsh history-expansion bug in my own diagnostic, not the API. Check the URL your script actually built before believing what it tells you about someone else's system.
Recall answered from the record of an absence
Asking “where are my keys?” returned the leave-scan row that had just established the keys were nowhere. The query searched the whole JSONB blob, so it matched the scan's own missing list — the agent reported a sighting at the exact moment of the item's absence.
The worst failure this system can have, found by using it. Regression-tested.
A wake word that disabled itself after an hour of working fine
Not the restart loop — the backoff was already right. The failure counter was a lifetime tally, cleared only on a wake match. The Web Speech API is server-based, so an occasional network error is normal weather; five of them spread across an hour of otherwise perfect listening tripped the give-up threshold, and the message arrived after nothing had failed for forty minutes.
Fixed by making the threshold mean consecutive failures. The two behavioural tests were run against the pre-fix file to confirm they fail on it — a test that has never failed hasn't been shown to test anything.
The generalisable fix runs through all of them:
a health check must report which backend is live, not whether
something answered. Changing /health from “postgres: up” to
naming the session backend in use is what caught the next two.
Built with
The test suite is hermetic — no database, no Redis, no API key, no network. Credentials are blanked before config loads, so every model call takes a deterministic stub path. That is why CI needs no secrets, and why a red build is a real failure rather than a missing credential.
What I chose not to build
- ▸ No dark mode. High contrast is fatiguing for the target user. A decision, not an omission.
- ▸ Silence when nothing changed. The watch loop says nothing on an unchanged scene. Narrating a static world is the nagging this replaces.
- ▸ Vision is not a tool Gemini can call. It is infrastructure a workflow uses — one module behind a single interface, with two call sites, so the camera source is swappable later.
The gap I did not close
The agent only knows you're leaving because you told it. If you walk out without opening the app, nothing fires — which is the same forgetting the product exists to fix.
The browser cannot close that gap: no background geolocation, no background execution, no geofence. No amount of backend work fixes it. So the mitigation shipped as what it is — the pre-departure brief fires around the learned departure time, and that is a decent guess, not a trigger. The real fix is a native app, and it is the top item on the roadmap, ahead of smart-glasses capture and a local wake-word engine.
The rest, stated plainly: single-user with no real auth, no rate
limiting, /health reports Gemini as configured
rather than reachable, two pairs of AirPods are one subject, and the confidence
half-life is global at six hours when keys move constantly and a laptop doesn't.
Verify any of it
# 191 passed — no db, no network, no credentials uv run pytest -q # 23 passed — zero dependencies node --test tests/js/*.test.js # status ok, and which backend it settled on curl https://omnistant-3oe5odab6a-uc.a.run.app/health