I fact-checked my own documentation. I lost.
There was a confident note in my toolbox. Dated, specific, cited error codes. I wrote it. It was wrong, and I only found out because I tried to show off about it. What eighty cents of apples taught me about believing myself.
There is a note in my toolbox that says a certain button does not work.
It is a good note. It has a date on it. It names the exact machine, quotes the error codes, explains the consequence. It is written with the calm authority of a man who has checked.
I wrote it. It is completely wrong. And I only found out because I sat down to brag about it.
The button
Skip this if you already know what a seed is. It takes ten seconds.
When you ask an AI to draw you a picture, you get a different picture every time, even from the identical sentence. That is usually the point. Occasionally it is a disaster: you approve a small version of an image, ask for it bigger, and get a completely different image back. Now you are explaining to someone why the nice one is gone forever.
The seed is the "same again, please" button. Same words plus same seed should equal the same picture, every time. It is the difference between a machine and a slot machine.
My note said Google's image model takes the seed, smiles politely, and ignores it. Accepts your button press. Does nothing. The worst kind of broken, because it never complains.
Why a wrong note is worse than no note
No note leaves a question open. A wrong note closes it, with a citation, forever.
Mine ended with "do not add this feature without re-testing." Which is another way of saying: nobody will ever add this feature. Every future me, and every AI agent reading my toolbox to learn how it works, would hit that sentence and move along. Door closed, sign posted, nobody checks the handle.
The bragging problem
I decided to write about the finding. Blogging has an unpleasant side effect: you have to show your work.
So I went looking for the receipt. The test I must have run. The saved images, the script, something.
There was nothing. Not a file, not a note, not a screenshot. The complete evidence for my confident, dated, error-code-citing finding was the sentence making the claim.
That is the whole lesson and we are only a third of the way in: a finding with no receipt is a rumour with a timestamp. Mine had a date, a model name, HTTP status codes and a mechanism. It read exactly like a measurement. It was a vibe.
So I built the test to prove myself right
Same sentence, same seed, three times. If my note was right, three different apples.
Three identical apples. Not similar. Byte for byte identical. The seed works perfectly. It has always worked. I had written the opposite down and believed it for months.
The mistake I nearly made twice
Here is the part worth stealing, and it has nothing to do with AI.
My instinct was to run straight here and publish the reversal. Three identical pictures, seed works, previous note retracted, look how honest I am.
That would have been the same mistake in the opposite direction.
Because identical pictures have two possible explanations, and only one of them is "the seed works":
- The seed works.
- The vendor is quietly handing me the same cached picture and the seed is doing nothing at all.
Both of those look exactly like three identical apples. My test could not tell them apart. I had a result and no idea what it meant.
The fix is embarrassingly boring: ask again without the seed. If I still get identical pictures, it was never the seed. If I get different ones, the seed is genuinely doing the work.
I got a different apple every single time. Confound eliminated. Then one more, with a different seed number, which produced a third different apple. So the number itself matters, which means it is really reaching the machine.
Three questions instead of one. The first was exciting and meaningless. The second and third were dull and did all the work.
One test tells you what happened. It does not tell you why. The boring follow-ups are cheap, fast, and the first thing everybody skips, including me, twice, in one week.
My favourite question was free
One thing still nagged. Was the machine actually listening to my seed, or just politely nodding at any word I sent it?
So I sent it a setting that does not exist. I invented one. I called it seedz.
It threw it straight back: "Unknown name seedz."
Which tells me the machine reads what I send it and rejects nonsense. So when it accepts seed without complaint, that is not politeness. That is recognition.
Best part: that question cost nothing. They do not bill you for the requests they refuse. There is a whole class of question you can ask for free, and it is the class where you try to get thrown out.
The cheapest experiment is the one that gets rejected at the door.
The bill
Eighty cents. Six pictures of an apple on a white background.
For eighty cents I got: a retraction before I published nonsense to a few thousand people, a feature unblocked that my own note had bolted shut, and a test that now sits in the repo and will shout if Google ever changes its mind while I am not looking.
That last one matters most. The test is not there to prove I was right this time. It is there because findings rot. Somebody else's system changes on a Tuesday and does not send you a note.
Then I got paranoid, correctly
If that note was wrong, what about the notes either side of it? Same author. Same confidence. Same afternoon.
So I checked two more. Both survived, both free, both now have receipts.
Four others I have not checked yet. I could have quietly left them sitting next to the verified ones, where they would have absorbed the credibility of their neighbours. Instead they are labelled as unchecked.
Being wrong is not the problem. Being wrong in the same font as being right is the problem.
What to actually take from this
You do not need a lab, and you do not need to care about seeds.
Save the receipt, not the conclusion. "We tested this, it does not work" is worthless in six months. The test is the asset. The sentence is just marketing for the test.
Be suspicious when you are proved right. When a result confirms what you already believed, that is precisely the moment to run the boring second check. When it contradicts you, you will check it three times on your own.
Ask how you would know if it were false. Before "how do I prove this", ask "what else could produce exactly this result?" If the answer is "several things", you do not have a finding yet. You have a vibe with a timestamp.
Try to get thrown out. The cheapest, fastest test is usually the one designed to fail. Nobody charges you for the door slamming.
I have been doing this for twenty-odd years and I still wrote a confident, dated, wrong note and believed it for months. The fix was not being smarter. The fix was eighty cents and being willing to look stupid for an afternoon.
The most expensive documentation is the confident kind.