Skip to content

What happens when this fails?

What happens under load nobody planned for?

What happens when the thing it depends on is already down?

How does it recover, and who finds out first — the user, or the on-call engineer?

Most systems are designed for the day they launch. I'm more interested in the day they break.

Principles

Held loosely. Still being tested.

  1. 01

    Reliability is a feature, not an afterthought.

    — I only really believe this because I once shipped something that wasn't.

  2. 02

    A system you can't observe in production is a system you don't actually understand.

    — I still guess more than I'd like to admit.

  3. 03

    Correctness before cleverness.

    — Cleverness is more fun. That's exactly the problem.

  4. 04

    Understanding before optimization.

    — I've optimized the wrong thing before. More than once.

  5. 05

    Every abstraction hides complexity somewhere else.

    — The job is knowing where. I don't always know where yet.

  6. 06

    Simple systems age better than clever ones.

    — Mine aren't simple yet either.

  7. 07

    There are rarely perfect solutions. Only trade-offs.

    — “Informed” is doing a lot of work in that sentence.

  8. 08

    Build the thing, not the demo of the thing.

    — Vectra is the proof. It's still in progress.

>break it into parts

>find the seams

>assume the seams will leak

>design for the leak, not the happy path

>still not sure i got the boundaries right

I think in systems, not components. I break large problems into smaller, independent parts — then spend most of my time on the communication between them, because that's where the real failures live.

Every decision trades one problem for another. I've stopped looking for the version without trade-offs.

Building

Vectra

Not a demo project pretending to be a product. An attempt at one.

Started as
An exercise — a way to go deeper into backend engineering than tutorials allow.
Became
Something I'm no longer comfortable calling a side project.
What it is
A distributed ride-hailing platform, built the way one would actually have to survive production.
Architecture
Independent services. Real network calls between them. Real ways for those calls to fail.
Stack
Go · PostgreSQL · MongoDB · Redis · Kafka · Docker · OpenTelemetry · Prometheus · Grafana · Tempo
Open question
Whether a couple of these services should really be one. I go back and forth.
State
Unfinished. Some of the diagrams are already wrong. I haven't fixed them yet.

Currently unresolved

I'd rather build the wrong thing and learn why than read about the right one.

  • Saga orchestration — still not convinced compensating transactions are worth it for every workflow.

  • Circuit breakers — tuning thresholds by feel right now. that should bother me more than it does.

  • Chaos engineering — haven't broken anything on purpose yet. i should have by now.

  • Kubernetes — learning it the slow way — by outgrowing what came before it.

  • Observability — i can measure more of the system than i can actually explain.

  • Production readiness — the checklist is easy. trusting it at 2am is not.

Notebook

unsorted. unanswered. unedited.

Why do retries make a system feel alive — right up until they make it feel like it's lying to you?

Every abstraction hides another abstraction. At some point you're just choosing which lie to trust.

A circuit breaker that never trips was never tested. One that trips constantly is a symptom wearing a costume.

Distributed systems don't fail like monoliths fail. They fail in the seams — the parts nobody was directly looking at.

Saga orchestration reads clean on a whiteboard. The compensating transaction for the compensating transaction is where it stops being clean.

When does architecture become the complexity it was supposed to remove? Still haven't found a clean answer.

Idon'tthinkaboutsoftwareascode.Ithinkaboutitasthingspeopledependonwithoutnoticing.

Thebestengineeringdisappears—quietly,underneatharidearriving,apaymentclearing.Itasksnothingback.

Curiositycompounds.Sodoesdiscipline.Neitherworksalone.

Future

I don't know exactly what I'm building toward yet.

I know it has to disappear the way good infrastructure does.

Utsav Kaushik