FlowLedger: making a manual deploy safe to repeat

Context

FlowLedger is a small invoicing product run by two founders. They wrote it themselves, deployed it themselves, and had reached the point where the deploy was the riskiest part of their week. There was no operations person, and hiring one was not on the table.

The problem

The failure was not the missing variable. It was that a healthy-looking service could be broken for half an hour with nobody able to tell. Most of this project exists to make that one class of problem hard to repeat.

Constraints

• Two founders, both writing product code. Anything I left behind had to be maintainable by people whose main job is not operations.
• One server. No cluster, no platform team, and no budget for either.
• An existing Jenkins box, already holding the credentials and the network access to deploy.
• Everything had to keep working after I stopped being involved.

My role

External contributor, working alongside the two founders. I did not write product features. I was responsible for how the product gets built, checked, shipped, rolled back and watched, and for writing that down so it survives without me.

Architecture

The runtime is deliberately small: the API with PostgreSQL and Redis, and an observability layer beside it that scrapes the application and routes alerts.

Key decisions

Every choice is recorded as a short architecture decision record, so the setup is a deliberate design the team can question later rather than something nobody dares touch.

Delivery

Two systems, two questions. GitLab CI decides whether a change may merge. Jenkins decides whether a build may ship. They are kept in step by the same pinned tools and the same configuration files.

Merge gate

GitLab CI, mirrored to GitHub Actions, on every merge request

Ship gate

Jenkins, staging first, then a human, then production

Implementation

• A reproducible environment. One command brings up the API, PostgreSQL and Redis in Docker Compose, each with a healthcheck. A /health liveness probe, and a /ready probe that returns 503 and names the dependency that is down.
• CI that gates every merge. Tests, image build, secret scanning with Gitleaks and vulnerability scanning with Trivy, mirrored to GitHub Actions so the public repository is gated too.
• A deploy path in version control. A declarative Jenkinsfile deploys to staging, runs a smoke test, then stops at a manual approval before production.
• Runbooks for the bad days. Written deploy and rollback runbooks, an incident template, and a lightweight change-management flow. Rollback is one command back to a previous tagged image.

Observability

Prometheus scrapes a /metrics endpoint, Grafana draws the dashboard, and Alertmanager routes four alerts, each chosen because it maps to something a customer would feel: the service being down, the 5xx rate climbing, the container restarting unexpectedly, and latency getting worse.

What went wrong on the way

The readiness probe was the interesting one. My first version reported ready as long as the process was up, which is exactly the blind spot that caused the outage in the first place. It only became useful once it checked the dependencies and returned 503 naming the one that was down.

The second was scope. It is tempting to hand a small team the setup you would build for a large one. Every decision here had to pass a simpler test: can two people who do not do this for a living keep it running.

Outcome

Detection

Minutes, not half an hourthrough the error-rate alert

Rollback

One commandto a previously tagged image

Operational alerts

Fourdown, 5xx, restart, latency

Delivery controls

Two gatesmerge in CI, ship in Jenkins

Every release is tagged, so a rollback is one command to a known-good version. Delivery became boring, reversible and written down.

What I learned

Health checks tell you the process is alive. They do not tell you the product works. The signal that would have caught this outage was the error rate customers were already experiencing, and no amount of green probes substitutes for it.

The other lesson was about documentation. The runbooks were not paperwork, they were the deliverable. A setup that only I can operate would have left the founders exactly where they started.

Technology

Runtime

Docker · Docker Compose · PostgreSQL · Redis

Delivery

GitLab CI · GitHub Actions · Jenkins · Gitleaks · Trivy

Observability

Prometheus · Grafana · Alertmanager

Case study note. FlowLedger is a pseudonym. The technical work and the incident are real. Identifying details, private business logic, hostnames and private code have been removed or generalised, and this write-up is published with the agreement of the team.