9 containers. 1 VM. No Datadog bill.SaaS pricing assumes you have units. I have a VM.One container grew a single 42 GB log file.The best commit of the migration deleted a container.Rent buys absence. Tuition compounds.Keep one system that fails at you personally.

Nine containers. One VM. About $61 a month, all in.

That's my entire observability stack: log aggregation, metrics, dashboards, alerting to Slack, uptime probes. I verified it this morning over IAP SSH: sudo docker ps on the VM shows exactly nine containers, every one of them up three weeks straight. No Datadog. No Grafana Cloud. No per-host anything.

It was ten containers until last month. Deleting the tenth is the best part of this story. I'll get there.

The problem, honestly stated

My fleet is small. Two Cloud Run services (this blog is one of them) plus the monitoring stack itself. I still want everything a real fleet wants: queryable logs, metrics with history, dashboards, alerts that find me in Slack, blackbox probes on the public endpoints.

SaaS observability pricing is shaped for someone else. Per host, per GB ingested, per user, per million events. The unit economics assume you have units. I have a VM and two services.

There is also a bias I'll admit up front. FinOps is part of my day job, and cost visibility is a reflex. The homepage of this site renders my live month-to-date Claude spend from a GraphQL API. If I meter my own AI tokens, I am not going to hand observability a blank check.

So the real question: what does the alternative actually cost, in dollars and in pain? Here is the full accounting, both currencies.

What $61 buys

The stack is called sentinel. One GCE VM, e2-standard-2 (2 vCPU, 8 GB RAM), 80 GB pd-balanced disk, Ubuntu 22.04, in us-central1-b. On it, nine containers: Grafana, Loki, Prometheus, Alertmanager, Alloy, node-exporter, Pushgateway, an nginx doing basic-auth in front of the Pushgateway, and blackbox-exporter.

The repo's own README estimates the monthly bill:

The whole thing is small enough to hold in one head. 678 lines of Terraform, 29 resources. 693 lines of stack config across 8 files. A 280-line bootstrap script. 10 Prometheus alert rules, 9 scrape jobs, 15 Grafana dashboards provisioned from JSON in git. You can grep the entire estate in an afternoon, and I mean that literally: every claim in this post is a grep away.

The mechanism: two data paths

Everything flows through one of two pipes.

Logs never touch the VM disk as storage. Services write JSON to stdout. Cloud Logging picks it up, a sink publishes to Pub/Sub, and Alloy (Grafana's successor to Promtail) consumes it with loki.source.gcplog and writes to Loki. Loki stores its chunks in GCS, so the VM only moves the data through. The 193-line Alloy config normalizes the service label across Cloud Run, Cloud Functions, GCE, and Cloud SQL log shapes at ingest time, from the __gcp_* resource metadata. Not from parsing log lines. Parse-at-ingest from metadata is boring and correct; regex-on-content breaks every time an upstream log shape drifts, which is often.

Metrics split by lifetime. Anything with a stable address gets scraped: 9 scrape jobs across the VM and the fleet. Anything ephemeral (Cloud Run, where the instance can be reaped mid-thought) pushes to the Pushgateway behind nginx basic-auth. The client libraries are mine and tiny: 388 lines of Go (canonical, flushes on SIGTERM so a Cloud Run reap doesn't eat the final datapoints) and 263 lines of Node. There was a Python lib. It's deleted. No new Python is a house rule.

On top: Grafana reads both stores, Prometheus evaluates the alert rules, Alertmanager routes everything (info, warning, critical) to Slack, split by severity into separate receivers. Slack is the pager here; a PagerDuty receiver stays unwired until this fleet earns one. Retention is 60 days on both Loki and Prometheus, set explicitly in both configs.

Cardinality gets the same discipline as cost. Labels stick to a short list: service, type, runtime, experiment, plus level on logs. Job IDs and request IDs stay in log content, never in labels. Cardinality is what actually drives metrics cost, self-hosted or SaaS.

The bootstrap is the real product

Terraform builds the shell. One idempotent bash script builds the stack: rsync configs from GCS, fetch secrets from Secret Manager, render templates, docker compose up -d. CI re-runs it after every infrastructure apply.

Two details in those 280 lines carry most of the operational weight.

First, it hashes each service's config directory before and after sync, and restarts only the services whose bind-mounted config actually changed. Compose won't recreate a container because a mounted file's content changed, so the script has to notice for it. It also skips services that up already recreated, to avoid doubling the observability gap. The helpers have their own unit tests.

Second, template rendering is allow-listed:

# ev-infra-group/infra/sentinel-vm/scripts/bootstrap.sh
# Allow-list only - unbounded envsubst would eat Prometheus / Alertmanager
# Go-template variables like $labels and $value.
SUBST_VARS='${SLACK_WEBHOOK_URL} ${GRAFANA_ADMIN_PASSWORD} ${PROJECT_ID} ${GCS_BUCKET_NAME}'

Run naked envsubst over a Prometheus alert rule and it will happily null out $labels and $value, and your alerts will fire with empty bodies. The allow-list is four variables long because four is how many it needs.

Security posture, briefly, because it shapes everything: the VM has no external IP. Cloud NAT for outbound, IAP-only SSH, and the UIs are IAP-fronted HTTPS behind the shared load balancer. There are no service account key files anywhere; auth is GCE metadata with cloud-platform scope. Secrets exist as Terraform objects, values added out-of-band so they never land in state, shell history, or argv.

What broke: the disk is yours to fill

Now the tuition payments.

2026-05-20. Two critical alerts. One: Grafana had been down for roughly two days, with 2,803 restarts on the counter. Two: the root filesystem was under 20% free.

The Grafana chain is a beautiful little horror. A consumer service got decommissioned and its Cloud SQL went with it, taking a Secret Manager secret (sentinel-reader-password) that the bootstrap still fetched. Under set -euo pipefail, the dead fetch aborted the script before the template render loop. Configs kept their raw ${VAR} placeholders. Grafana read an empty Slack URL and crash-looped on "token must be specified." For two days. 2,803 times.

The disk chain is dumber and better. The compose file had no logging: blocks and the daemon had no daemon.json, so Docker's json-file driver kept one unbounded log per container (rotation is off by default). The Pushgateway, when pushers send inconsistent HELP strings, dumps the entire offending metric family to stderr on every scrape. That's roughly 100 KB per line, every ~15 seconds, around the clock. One container had grown a single 42 GB log file.

?Why this image

The fix is ten lines and a flag:

# ev-infra-group/infra/sentinel-vm/docker-compose.yml
x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"

Every service gets logging: *default-logging. The comment above the anchor says why: without it a chatty container "silently fills the VM disk." The anchor "Caps each container at 3 x 10 MB." Pushgateway also got --log.level=warn. The commit records the disk going from 63 GB used to 20 GB.

A managed backend absorbs this entire class of failure invisibly. You pay for that absorption whether or not you ever need it. Self-hosting refunds the money and hands you the failure. The disk is yours to fill is the honest one-line summary of the whole model.

The flag that killed Prometheus

Second payment. A hardening PR locked down Prometheus's lifecycle endpoints with --web.enable-lifecycle=false. Reasonable syntax. Wrong parser. Prometheus uses kingpin boolean flags: --no-<flag> disables. The string "false" parsed as a positional argument and the process died with unexpected false. Crash-loop from the apply at 04:51 UTC until the fix landed.

# ev-infra-group/infra/sentinel-vm/docker-compose.yml (prometheus command)
      # Prometheus uses kingpin boolean flags: `--<flag>` enables,
      # `--no-<flag>` disables.
      - --no-web.enable-lifecycle
      - --no-web.enable-admin-api

The lockdown itself was not paranoia. IAP gates who gets in, but Prometheus has no per-user authorization. Any admitted user could otherwise POST /-/reload or /-/quit. Defense in depth found a syntax landmine, the landmine got a comment, and the comment now guards everyone who touches that file.

The tenth container

On 2026-06-23 the whole stack migrated projects: old foundation out, new one in, different region, parallel-run with no forced cutover. The old compose file ran ten containers. The tenth was a Cloud SQL Auth Proxy, there for a database that does not exist in the new project.

The lazy migration carries the compose file as-is. The repo's ADR refused, with mechanism: a carried proxy would crash-loop against a database that isn't there, and dead scrape targets would fail the disaster-recovery drill's health check and fire false alerts. So the migration stripped the dead config instead of carrying it. The CHANGELOG states it flatly: "10 → 9 containers."

?Why this image

The full container arc is the health signal: 7 at v0.1.0 (tagged one day after the initial commit), 10 at peak, 9 now. A stack whose container count can go down during a migration is a stack someone actually reads.

One footnote from that migration, for flavor: the IAP OAuth brand had to be configured by hand in the console, because Google retired the gcloud API for it three months earlier. Twenty-nine Terraform resources, a five-step cross-state apply sequence, and one un-scriptable console click. Self-hosting on a cloud means you also inherit the cloud's retirements.

Rebuild, don't nurse

My name for the DR posture: rebuild, don't nurse. The VM is cattle with one pinned property.

Everything that matters survives a VM replace. Loki chunks live in GCS. Dashboards are provisioned from git. Secrets are in Secret Manager. And the internal IP is pinned so a rebuild is invisible to everything pointing at it:

# infra-sentinel/terraform/vm.tf
  # Pinned so the reservation can't drift to a different IP if this
  # resource is ever recreated. Service repos and firewall rules
  # already assume 10.3.0.2 (the .2 host of the infra-sentinel registry
  # slot, 10.3.0.0/24); an unannounced reassignment would be a quiet
  # outage.
  address = "10.3.0.2"

  lifecycle {
    prevent_destroy = true
  }

The accepted loss is explicit: up to 60 days of Prometheus TSDB history. Disposable, by decision, in writing. A 245-line rebuild-from-scratch.sh drives terraform apply -replace=, and it refuses to run from a dirty checkout or a non-main branch, so a rebuild can't smuggle unrelated infrastructure edits in under the cover of an emergency.

And because a monitoring stack that monitors itself is a liar with one witness, there is a watchdog for the watchdog: the bootstrap installs the Google Cloud Ops Agent so disk, CPU, and memory land in Cloud Monitoring independently, and out-of-band Cloud Monitoring alerts (added at the June migration) cover the one alert this stack can never send: "the stack itself is down." The May incident tested a nearby seam. Grafana was dead for two days, yet both alerts arrived on time, because the alert path is Prometheus rules through Alertmanager into Slack and never touches Grafana. The display layer died; the alarm layer kept working. The out-of-band layer exists for the uglier day when Prometheus or Alertmanager is the corpse.

?Why this image

Rent versus tuition

Here is the frame I now use for every build-vs-buy call in this category. SaaS observability is rent. Self-hosting is tuition.

Rent is smooth and predictable, and it buys absence: absence of the 42 GB file, of kingpin syntax, of your own bootstrap's failure modes. Rent is the correct product for most companies, most teams, most on-call rotations. But rent leaves nothing behind. Stop paying and you hold exactly what you held on day one.

Tuition arrives lumpy and unscheduled: a full disk, a 2,803-restart counter, a crash-loop at 04:51 UTC. Every payment converts into an asset with a git blame, though. A logging anchor. A four-variable allow-list. A comment that stops the next person from shipping =false to a kingpin binary. Rent stops the day you stop paying. Tuition compounds.

The dollars matter less than people pretend. $61 a month is the number to beat; take whatever SaaS quote covers logs plus metrics plus dashboards plus alerting for your fleet and run the comparison yourself. For me the deciding currency was the other one. I manage platform and infrastructure teams for a living. If my only contact with failure modes is reading my teams' postmortems, that muscle atrophies. This stack is my gym membership, and unlike most gym memberships, it pings me when I skip.

Should you copy this?

Only if the tuition is the point. The repo states its own tradeoffs, so I'll state them louder:

• The VM is a SPOF. Single instance, no managed instance group. A rebuild costs up to 60 days of metrics history. The repo's own docs say to re-evaluate before adopting this for production.
• You own upgrades. Nine pinned image tags, bumped by hand via a runbook. Nobody upgrades Grafana for you at 3 AM. Nobody CVE-patches Loki for you either.
• You own every failure mode, including your own tooling's. The worst outage in this stack's history was not caused by Prometheus or Loki. It was caused by the bootstrap script written to manage them.
• Coordination is real. Infrastructure and stack live in two repos with a sequenced cross-state apply ordering. Small, but it is process you maintain.

If you run a product team, a real on-call, and revenue-bearing SLOs: pay rent, and pay it happily. If you run a small fleet, want the sharpest possible view of what observability actually is under the vendor shrink-wrap, and can afford to lose two days of Grafana without losing money: 678 lines of Terraform and 693 lines of config is the entire syllabus.

I learned to program by breaking machines I owned, starting with a C64. More than two decades into this career, the method has not changed, only the machines got rented. This VM is one of the few things left in my infrastructure that fails at me, personally, with no support ticket to hide behind. Nine containers, one VM, and every scar in the git history is mine.

I intend to always keep one system around that charges tuition. Right now, it's this one.