There's a new kind of tool showing up this year: an AI that watches your session recordings for you. You point it at your replays, describe what you're worried about, and a model watches the tapes, flagging bugs, scoring frustration, tagging sessions, summarizing what it saw.

I want to say up front: I get the appeal, and part of it is genuinely right. The premise of these tools is that nobody watches session recordings, and that premise is correct. Everyone records everything and watches almost nothing. We built a whole product on the same observation. So when the pitch is “the recordings have the answers and you've never had time to look,” I agree with every word of it.

Where I get off the train is what happens next.

When a model watches a recording, what you get back is a verdict and a hypothesis. This user seemed frustrated. This session looks like an abandoned checkout, probably about price. That's not nothing, it's a fast, tireless intern giving you their read on ten thousand sessions. For triage, that's useful.

But a read is not a measurement. If the question is “did checkout completion drop after Tuesday's deploy,” an opinion, even ten thousand opinions, is the wrong shape of answer. You need the same cohort, the same windows, the same funnel, computed the same way on both sides of the release, and a straight answer about whether the difference is real or noise. That's arithmetic, not vibes. A model that watched the video can't give you that number, because it never computed one. It watched.

This is why we've been strict about one line since the beginning: every figure is computed from sessions, never guessed by a model. The model reads your PR description to work out what you claimed. It never writes a number.

The second problem is quieter. These tools ask you to describe what to look for. The scanner is a prompt: “flag users who hit the paywall and bounce.” “Tag abandoned checkouts by reason.”

A prompt like that is a spec. You wrote it from what you already suspect, it finds what you asked about, and it misses everything you didn't think to ask. Nothing in a prompt knows that four thousand returning users reach for one particular button without looking at it, unless you knew that, and you didn't, which is the entire problem.

And prompts rot. I wrote a few weeks ago about loop engineering, how the artifacts you reuse (rubrics, checklists, specs) quietly drift out of date because nothing is watching them. A library of scanner prompts is exactly such an artifact. Six months from now it's a museum of things you worried about in February.

The alternative isn't a better prompt. It's not asking a human to describe what matters at all. A behavioral load map is computed from the sessions themselves, which elements carry completions, which paths returning users run on muscle memory. Nobody writes it, so it can't drift from reality. It regenerates from reality.

The third problem is the one I keep coming back to, because it decides whether any of this is still installed in a year: none of these findings ever find out whether they were right.

A model flags a session as “frustrated, probably the coupon field.” Fine, was it? Did anything change when the coupon field got fixed? Did the finding predict a real behavior shift, or did it just sound plausible? If nothing ever grades the finding, every scanner stays exactly as trustworthy as the day it was written, forever. Which, as anyone who has muted a noisy bot knows, means less and less trustworthy in practice, while staying just as loud.

We measure what every release actually does to user behavior. That's the product we started with, and it's why our PR-time predictions can be graded against what really happened and demoted when they're wrong. The grading isn't a feature bolted onto the analysis. It's the reason to believe the analysis.

So: should you use an AI that watches your sessions? Honestly? Maybe. As a summarizer of weird individual sessions, as a triage layer over a pile of replays you were never going to watch, it's a real convenience, and I expect these tools to get good at it.

Just keep asking the question that separates an observer from an instrument: how does it know when it was right? If the answer is that nothing ever checks, you have a witness. Witnesses are useful. But when the numbers matter, when it's your checkout, your release, your Thursday merge, you want the measurement.