Your Fallbacks Are Hiding Your Bugs

I shipped a feature that was already working on my laptop, watched the deploy go green, and only found out days later that the feature had never run in production at all.

Nothing was broken. That was the problem.

I built an agent called Omnistant for the All Things Agentic Hackathon— it watches a camera feed, remembers where your things were last seen, and tells you what you're about to walk out without. Five real bugs came out of that build. I want to write about them together, because they turned out to be the same bug wearing five different costumes.

Every one of them failed quietly. And every one of them failed quietly for the
same reason: the subsystem in question had been carefully designed to degrade rather than crash.

The deploy that was green and wrong

Conversations in Omnistant are stored in Postgres, through Google ADK's
DatabaseSessionService. The alternative, InMemorySessionService, keeps the whole conversation in one process's RAM; a restart erases it, and on more than one instance, turn two can land on a machine that never heard turn one. On Cloud Run, which scales to zero, in-memory sessions are not a session store. They are a way of losing conversations.

So I wired up the Postgres-backed one. It needed sqlalchemy and greenlet. I
ran uv add, tested locally, watched it work, and deployed.

What I did not do was regenerate requirements.txt, which is what the Dockerfile actually installs. pyproject.toml had the new dependencies. The deployed image did not.

Here is the part that cost me the days: the service came up perfectly. ADK
tries the database backend, catches the ImportError, falls back to in-memory,
logs a line nobody was reading, and serves traffic. Health check: 200. Container: healthy. Feature: absent.

That fallback is correct behaviour. I want a session store that degrades instead
of refusing to boot. But it converted a loud failure — ModuleNotFoundError,
crash loop, obvious into a state change that nothing was watching.

Four more of the same shape

Once I had the pattern, I recognised it everywhere in the build.

Recall answered from the record of an absence. Item lookup searched the observation log with content::text ILIKE '%term%' — a substring match against the whole JSONB blob. But a "leaving home" scan stores the items that were missing:

{"destination": "work", "missing": ["keys", "wallet"], "found": []}

So asking "where are my keys?" matched that row and reported a sighting — at
precisely the moment the agent had established the keys were nowhere. No error. No empty result. A confident, well-formatted, exactly-wrong answer, which is the worst kind an assistant can give you.

Answering out loud killed the microphone. Speech synthesis and speech
recognition compete for the audio device. Start recognition while the page is
speaking and it ends instantly — with no error event. The failing path was the
conversational one: the agent answers aloud, you tap the mic to reply, the mic
dies. It only broke when the interaction was working as intended.

The wake word never fired. Two reasons, and the second is the one I'd have sworn wasn't possible. First, "Omni" is an invented word, so speech recognition never returns it the same way twice — omny, omnie, on me, omani are all transcriptions of the same sound, and matching an exact string means the wake word simply does not work. Fine, my mistake, fixable with a pattern:

WakeWord.DEFAULT_PATTERN =
  /\b(?:hey|hi|hello|ok|okay|yo)\s+(?:omni\w*|omn\w+|omany|ohmni|amani|on\s?me|almighty)\b|\bomnistant\b/;

Second: normalisation stripped punctuation by replacing it with a space. "Hey, Omni!" — one of the most likely things a person says — became "hey omni" with two spaces, and the match failed on a string where every word was correct. One character, invisible in logs, in the highest-traffic phrasing there is.

var heard = String(text)
  .toLowerCase()
  .replace(/[^a-z\s]/g, ' ')
  .replace(/\s+/g, ' ')   // this line is the fix
  .trim();

And then, while writing this article, I found the fifth one. I went to verify
the test count in my README before quoting it. The README said:

uv run pytest -q          # 83 tests, ~0.6s

I ran it. Four modules failed to collect: ModuleNotFoundError: No module named 'tests'.

Four test files import from tests.conftest import .... My pytest config put
src on the path but not the repo root. python -m pytest silently inserts the
current directory into sys.path; a bare pytest does not. I had been running
it one way. The README documented the other. The suite passed for me every single time and was uncollectable for anyone following my own instructions.

The count was wrong too — 83 was stale. It was 105.

The actual pattern

Look at what these five have in common. It isn't a language, a layer, or a
category of mistake. Two are backend, two are browser, one is packaging.

In every case, a subsystem was built to keep working when something underneath it wasn't available — fall back to in-memory, match loosely, restart the recogniser, search broadly, insert a path entry. Every one of those decisions is defensible. Most of them I'd make again.

And every one of them takes a failure that would have been loud and makes it quiet. That's not a side effect. That is definitionally what graceful
degradation does: it trades a crash for a degraded state. The crash was your
notification. If you don't replace it with something, you have removed your only
alarm and installed nothing.

The bug isn't the fallback. The bug is falling back without telling anyone.

What actually fixed it

Not "add more logging". Logs are where this class of bug goes to be ignored — the in-memory fallback did log, and I never saw it.

The fix was changing what "healthy" is allowed to mean.

My health endpoint used to answer the question did something respond? Every one of these bugs answers yes to that. So I made it answer a different question: which implementation is actually serving?

def state(healthy: bool, kind: str, real: str) -> str:
    if not healthy:
        return "down"
    return "up" if kind == real else f"fallback ({kind})"

postgres = state(store_ok, store.kind, "postgres")
redis = state(cache_ok, cache.kind, "redis")

# A fallback is not a failure, but it is not full health either: nothing
# written to an in-memory store survives the next cold start.
degraded = not (store_ok and cache_ok) or "fallback" in postgres

return {
    "status": "degraded" if degraded else "ok",
    "postgres": postgres,
    "redis": redis,
    "sessions": get_engine().session_backend,
    "gemini": "configured" if cfg.genai_available else "missing credentials",
    ...
}

"sessions": "in_memory" on a production instance is now visible in one curl.
The identical bug today takes about four seconds to find.

The generalisation I'd offer: if a component can degrade, its health check has
to name the mode it's in. A boolean is the wrong shape. up is a lie that a
fallback tells fluently — the whole point of the fallback is that it can answer
up. Report the backend, not the response.

The same reasoning covers the packaging bug, which no runtime check would ever catch, because the fallback is genuinely healthy. That one needed a test that compares what pyproject.toml declares against what requirements.txt pins:

@pytest.mark.parametrize("package", [
    "google-adk",   # the agent runtime
    "sqlalchemy",   # ADK DatabaseSessionService
    "greenlet",     # SQLAlchemy's async bridge; missing = silent fallback
    ...
])
def test_runtime_critical_packages_are_pinned(package):
    """Each of these, if absent, degrades something quietly instead of crashing."""
    assert _normalise(package) in _pinned_in_requirements()

All five now have regression tests, including the pytest path one, which asserts
its own config:

def test_pytest_pythonpath_makes_the_suite_collectable_either_way():
    pythonpath = data["tool"]["pytest"]["ini_options"]["pythonpath"]
    assert "." in pythonpath
    assert "src" in pythonpath

The part I'd tell myself at the start

I don't think the lesson is "degrade less gracefully". A camera agent that
refuses to boot because Redis is missing is a worse product than one that keeps answering questions from Postgres.

The lesson is that graceful degradation moves the cost rather than removing
it. It buys you uptime and it charges you observability, and if you only do the
first half you've built a system that is very good at looking fine. Three of
these five bugs were sitting in code I'd have described as working. One of them was in the instructions I was about to hand to a judge.

So: every fallback path gets a name, and every name gets surfaced somewhere a human looks in the first thirty seconds. Health endpoints report which, never whether.

And run your own README. Out loud, from a clean clone, exactly as written. I had five bugs, and the fifth one was hiding in the paragraph telling everyone else how to check for bugs.

---

I built Omnistant and wrote this piece for the purposes of entering the #AllThingsAgentic Hackathon. The code is open source: github.com/TheGh0xt/Omnistant