QA automation

This is my day job. It is also the shelf that has taught me the least — and that turns out not to be a bad thing.

First, what “QA automation” is.

When you build a game or an app, someone has to press everything and check it works. Press a button, see if the right screen comes up, confirm the value landed. A person doing that by hand is QA. Getting a machine to do the pressing is QA automation.

There are six pieces here, each one someone else’s automation, opened up.

My own field, and the smallest harvest — which is not a complaint

Across the first five pieces, the techniques I wrote down as adopted number 4.5. Fewer than any other shelf. (Only one had actually gone in when I checked — see the 2026-09 check below.)

My first thought was I must be reading them badly. That wasn’t it. Hold the same problem long enough and everyone arrives at similar answers.

Put it this way: when there are only a few routes up a mountain, the person who climbed first and the person who climbed later meet at the same places. At that point the finding is not nothing to learn — it is my route was right.

Why the research still pays when nothing transfers

This is the important part of the shelf. Two different kinds of return came out.

One — a conclusion I held alone became a conclusion held by two. Some judgements I had made on my own, with no way to know whether they were right. Then someone I have never met, at a different company, had arrived independently at exactly the same answer. A sample of one became a sample of two. Even with zero techniques transferred, that is worth a lot.

Two — someone else’s principle became a ruler for my own thing. In one piece the transferable technique count was zero. But taking the principle they had set and measuring my code against it, I found a defence I had built was not wired into any actual run. Nothing learned, and something to fix.

“Already there” did not always mean the same thing

At two pieces I read it as one thing: I already arrived at this. At four, two different states were hiding under that phrase. Having arrived at the same answer and having written the answer down without ever doing it are not the same.

And I was caught by the second one — something I believed I had done turned out to be something I had written down as intending to do.

One permission that changed the nature of the evidence

There is a harvest here that isn’t a technique.

Checking whether a game’s drop rate matches what was advertised used to mean pulling hundreds of times and doing statistics. Slow, and the answer comes out as probably right.

When access to the underlying data arrived, the same check became one query, five seconds. And the answer stopped being probably and became certain.

→ What I took: it wasn’t a faster tool. The vantage point changed, and that changed the kind of evidence available.

In the fifth, I went to judge a detector and got caught by it

The first four were other people’s talks and articles. The fifth was the first time I pulled someone’s code and ran it against my own data.

I went to test whether their detector caught what it claimed. Run against my data, it flagged a pile of things that were fine. Not because the tool is bad — because it was the wrong tool for the place I meant to put it. Without measuring, it would have come straight in.

Of the 4.5 techniques I wrote down as adopted, one reached the code — a 2026-09 check

About a month on, I opened my code to see whether what each piece said it would take had actually gone in.

Piece	What it said I’d take	As of 2026-09
Asleep’s QA automation	Count passes that came from a rough coordinate tap separately	In — a tap ledger that records the evidence behind every tap
An autoplay test tool	Catch “pushing but not moving” as being stuck · confirm only when two clocks ring together	Not in at the check → both went in (9/28)
watermarks-remover	One sentence-rhythm term · a check that stops on an unknown kind of file	Neither was in at the check → both went in: the stopping check (9/28) and the rhythm term (9/29, long-form only)
Toss Nebula	Fix the “gateway that exists twice” it surfaced instead of a technique	Half — the real run path now has a lock; the gateway still exists twice
A talk on QA infrastructure	Use the gacha case as an example in writing	In — “Probably right” is not a verdict

Writing “adopt” and putting it in turned out to be separate jobs. Marking a row “adopted” in a verdict table feels like the work is done at that moment, and the code knows nothing about it. So this shelf’s number is now written as “4.5 of 4.5 applied” rather than “4.5 techniques” — the one that was in at the check, plus the 3.5 put in over the next two days (a publish check that stops on an unknown kind of file · the contradiction check for being stuck · the two-clocks confirmation · a long-form-only sentence-rhythm check). The effects differ. The rhythm check caught two easy-layer passages on its first run and they were fixed; the two stuck-detection pieces haven’t filtered anything in simulation yet — the current bot pushes the whole time it walks, and nothing in the simulation inches forward — so what went in and what has had an effect are counted separately too.

One mistake ran the other way. In the autoplay piece I said my stuck detection had only one layer; opening the code, a second layer was already there — that piece now says so.

And outside this shelf, something built by someone else entered my QA pipeline for the first time: the evaluation model Jev, attached as the photo judge’s “second eye” (2026-09-24). It came in through a four-day trial rather than a teardown, and in the range measured it has not yet caught a real error → The photo judge.

The sixth piece took the same subject apart twice and never ran it

clumsy is a tool that makes a PC’s internet connection worse on purpose. I opened it for disconnect testing, and only after finishing did I find a report from August that had already reached the same conclusion. Both times I wrote “use it”. Neither time did I run it.

The check above taught me that writing something down and putting it in are separate jobs; this time I went through that twice inside a single subject. So the shelf’s number becomes “4.5 of 7 applied” — clumsy adds 2.5 to what was written down, and nothing to what went in.

The detailed record starts here

Where other people’s test automation gets taken apart: device farms, unattended regression on real hardware, game autoplay tools, agent-driven test platforms.

It’s my day job and it yields the least. Across five pieces, 4.5 techniques transferred (as of 2026-09-29: 4.5 written into verdict tables · 4.5 in the code — 1 at the check, 3.5 after · see the 2026-09 check above). Not because the teardowns were bad — because enough years under the same constraints tend to land you in the same place already.

Which changes what this bucket is worth

Subject	Techniques	What actually stayed
Asleep’s QA automation	1 of 11	That under the same constraints someone else arrived independently at the same decisions
Toss Nebula	0	Using their principle as a ruler showed that my own gateway exists twice
An autoplay test tool	2 of 4	Catch being stuck as a contradiction in the data, and clear false positives with an AND over two timers
A talk on QA infrastructure	0	Not a tool but a case where the character of the evidence is decided by access
watermarks-remover	1.5 of 4	Measure a detector’s false-positive rate on your own corpus before adopting it — and on the corpus it will actually run against
clumsy	2.5 of 6 (0 applied)	Analysing the same subject twice left the verdict exactly where it was — only a measurement moves a verdict

An investigation that yields nothing isn’t wasted. Asleep turned a set of conclusions with a sample size of one into two — on-prem, fail-closed on live, unattended regression, element-based targeting. Nebula handed over zero techniques and instead surfaced that the official gateway’s defences aren’t wired into the path a real run takes.

And in both of the early ones, my own verdict got overturned

Asleep took two corrections. I wrote down “silence detection is a new harvest,” then deleted it because it was already in my code; then tried to close at “zero new,” and found that my excerpt had been too thin — the section the author calls the core wasn’t in it at all.

Nebula lost the only harvest in its first draft. I’d adopted an in-device controller on the strength of a real measurement — uiautomator dump at 2,350 ms — and then found that call appears zero times in my pipeline. A game client draws the whole screen into one native surface, so there is no accessibility tree to dump.

The measurement was right and the target was wrong. Before benchmarking anything, check whether the call actually happens in my run.

At four pieces, “already there” split into two meanings

At two I read it as one thing — “I’ve already reached the same conclusion.” At four, two different things turned out to be mixed together inside that.

One is genuinely the same place — ten of Asleep’s eleven decisions were that. The other looked like the same place and was one layer shallower. The autoplay piece was that. I was also judging when a bot is stuck, and I wasn’t wrong, I had simply never separated the layer that produces candidates from the layer that removes false positives.

You cannot tell these two apart without opening the thing, because both look like “we already do that.” The moment I split the layers, the cause of my false positives acquired a name — “a normal state where it stopped on purpose.”

(Correction, as of 2026-09) “Never separated the layers” was wrong for the autoplay piece. Since the July code, my stuck detection has had a second layer that confirms by trying to escape, behind the candidate (under 60 cm of movement in 3 seconds). What was one layer shallower is not the number of layers but the contradiction check that looks at input and movement together. That went in on 2026-09-28.

And this bucket holds one harvest that isn’t a tool

The infrastructure talk yielded zero techniques and left one case: verifying gacha probability. What used to be hundreds of pulls and a statistical estimate became one query in five seconds once read access existed — and the character of the evidence changed from estimate to certainty.

It didn’t get faster. What counts as evidence changed. It goes straight into the writing about verdicts, which makes it the only harvest from this bucket that turns into prose. (As of 2026-09 it went in: “Probably right” is not a verdict uses this case as a jump of several rows “from a sample to a direct read of the source”.)

The fifth was this bucket’s first repository, and I went to judge a detector and got caught by it

The first four were all somebody’s talk or article. watermarks-remover is the first piece here where I read code, which let me do the one thing only that allows — run someone else’s detector against my own data.

The result was the most expensive thing this bucket has produced. A metric built to separate AI writing flagged, on its first outing, something a human wrote — me. Sentence-length coefficient of variation across 98 long pieces on this site sits at a comfortable median of 0.668; move it to the drafts from the channel where the rule would actually live, and one in six falls under the threshold — and that one was already published.

Measure a false-positive rate in the wrong place and the rate itself is a lie. Calibrate on long-form, apply to short-form, and you are worse off than before you measured.

Read as QA this is a familiar failure: test data that doesn’t match the production distribution. Except this time I was the one who chose the test data. The Nebula piece taught me to ask “does this call actually happen in my runs”; this was the hole one seat over — the call does happen, but the sample I measured came from somewhere else.