WorkWritingAboutContactGet in touch
All writing
4 min read

Measure your AI tools, don't argue about them

Everyone has a loud opinion on whether AI coding tools help. Almost nobody has the data. So I built a tiny thing to get it, used it, and killed it.

Every team I'm near is having the same argument. Is Claude Code (or Codex, or Gemini) actually making us faster, or does it just feel that way?

Strong opinions on every side. Data on none. Everyone is very confident, which is interesting, because nobody is measuring anything.

I have no patience for arguing from vibes. So I built something to settle it.

It's called stark-insights. One Go binary, that's the whole product. Claude Code hooks fire every event into a local SQLite buffer, and a timer drains it to BigQuery every 15 minutes. No platform, no roadmap, no Slack channel named after it.

The stark-insights capture pipeline: every Claude Code, Codex, and Gemini hook fires one event into a local SQLite write-ahead buffer, which a launchd timer drains into BigQuery every 15 minutes for retros and ad-hoc queries.The stark-insights capture pipeline: every Claude Code, Codex, and Gemini hook fires one event into a local SQLite write-ahead buffer, which a launchd timer drains into BigQuery every 15 minutes for retros and ad-hoc queries.
Capture pipeline - hooks to a local buffer, drained to BigQuery every 15 minutes.

What it cost to run: almost nothing. A rounding error of BigQuery storage, and a hard $2.50 cap per query so a fat-fingered SELECT * couldn't quietly bill me a car. The expensive part of AI was never the telemetry.

The expensive part is the tokens

And the bill arrives with a personality.

Exhibit A, a confession I had to make: an auth error, every instinct in the building screaming OAuth problem, so naturally that is what got debugged. All day. Across multiple repos. ~$500 of tokens. It was a one-line load-balancer cert setting the entire time. The error code's own documentation said exactly that. Nobody read it.

Exhibit B: one agent job left running on a laptop, quietly burning ~$150 an hour to finish 40 of 200 URLs. We moved it to a VM before it bought itself a boat.

You feel none of this in the moment. That's the whole problem. The meter only exists if you build it.

What the data actually said

Over 60 days it logged 612 sessions across 15 projects running in parallel. A few things jumped out, and not all of them were flattering.

  • My median session went from 21 minutes to five hours. Deep flow, or I let the agent run while I made coffee. Honest answer: yes.
  • The multi-agent review pipeline threw 3,087 findings in two weeks. Claude 1,719, Codex 1,306, Gemini 62. Gemini also ran 86% noise on small PRs, so it found almost nothing and complained the most.
  • The workflows I was sure were paying off mostly were. A couple I defended out of habit were not. Those got cut.

Every few days I'd sit with the numbers and run a small retrospective on myself. Not a dashboard to admire. A question to answer, then close.

The playground lifecycle as a loop: build it once, then measure, retro, and ask what changed. If it hasn't answered yet, loop back to measure. Once it has, rip it out.The playground lifecycle as a loop: build it once, then measure, retro, and ask what changed. If it hasn't answered yet, loop back to measure. Once it has, rip it out.
The loop - measure, retro, repeat, then rip it out the moment it has answered the question.

The part everyone skips

When it had answered the questions, I killed it. Hooks out, jobs deleted, infra torn down, data archived, nothing left running. One afternoon. No farewell post.

A playground earns its keep by answering a question, not by surviving. The moment it stops earning, it's a pet you have to feed.

A playground earns its keep by answering one question. After that, it's just a pet you have to feed.

What this is not

Not a pitch to go build an observability platform for your AI usage. Please don't. Give it a logo and it's a product, products need owners, and congratulations, you now have a second job.

And not a license to wire hooks into your team. I measured myself, on my own machine, by choice. Pointing the same thing at other people opens with their consent, not your curiosity.

The bottom line

The vibes were mostly right. The few places they weren't are exactly where the money was leaking, which is the part that should keep you up at night.

Gut is a fine compass and a useless meter. Tokens don't care how confident you are. So measure the thing - cheaply, locally, only until it answers you - then throw the meter away. You don't need a platform. You need an afternoon and the discipline to delete it.

Then, by all means, have the opinion. You'll have earned it.

Back to all writing