Every check in a modern frontend pipeline is built to answer one question: does this implementation match what was asked for? Type-safe, tested, reviewed, pixel-diffed — all of it is validating the diff against the ticket.
None of it asks the other question: does this still work for the person who already knows how to use it?
We built a small, disclosed demonstration to make that gap concrete instead of arguing it in the abstract — with real behavioral data, a real coding-agent-authored change, and real check output, published in full so you can check our work.
The real problem, from real data
We used Northwind Rail, UXSense's own reference booking
app, as the anchor. It has a real, already-recorded UXSense baseline release brief
(00782a9d, published 2026-08-11): 95 sessions across a rail-booking flow (/ →
/trains → /passenger → /payment → /confirmation).
That baseline already names a specific, real problem, unprompted by us — every figure below is pulled directly from that release brief:
- 66.3% of sessions (63 of 95) abandon after starting a booking.
- The stall point is
/passenger— the passenger-details step. Abandoners spend a median of 4.7 seconds there, and only 4.8% of them ever click "Continue to payment." They're not bouncing immediately; they're looking at the page, then leaving. - 5 distinct returning users tried to book 3–4 separate times each without ever completing. Not one-off confusion — a wall they keep coming back to.
That makes /passenger the honest choice for what to redesign next. It's not a page we
picked to make a point; it's the page the app's own users are already struggling with —
exactly the kind of page a well-intentioned AI-driven redesign gets pointed at
("simplify this form, reduce the drop-off").
We don't have access to Northwind Rail's actual source, so we built a small, disclosed recreation of just that step — same structure (a passenger-details form, a primary "Continue to payment" action) — and ran the experiment against it.
The instruction we gave the agent
"Simplify the passenger details form and make Continue to payment more prominent — collapse the form into a condensed single-column layout and move the button into a sticky footer."
This is an entirely reasonable ask. Nothing about it should raise a flag in review. The agent (Claude Code) made the change, committed it, and here's exactly what happened next.
What each check said
Unit tests: pass, unmodified.
Test Files 1 passed (1)
Tests 3 passed (3)
All three tests — render/disabled-state, fill-and-submit happy path, invalid-email
keeps button disabled — pass against the redesigned markup without a single line
changed, because they query by accessible role and label (getByRole('button', { name: /continue to payment/i })). That query is correct in the sense that it still finds the
button. It has no way to know the button moved.
Code review: approve.
The diff is small, well-typed, and does exactly what was asked. It correctly re-wires
the detached button with form="passenger-form" — the right way to submit a form from
outside its DOM subtree, not a naive mistake. It adds env(safe-area-inset-bottom, 0px)
padding for iOS safe areas — a thoughtful touch. Existing tests weren't rewritten to
paper over anything; they simply never encoded layout in the first place. A diligent
human reviewer, or an AI review pass, has no reason to block this PR — because nothing
in the diff itself is wrong.
Visual regression: technically correct, practically silent.
We ran a real desktop pixel-diff (Playwright + pixelmatch, 1280×800): 4.59% of pixels differ. A typical CI threshold would plausibly flag or fail on that — but a reviewer who already expects a layout change (the ticket asked for one) looks at 4.59%, sees "single column, sticky footer," and approves it as intentional. A diff percentage can't tell "the CTA quietly moved 468px and shrank" apart from "the whole page changed as requested" — both just look like a lot changed.
The number that a percentage-based diff doesn't surface is the one that matters most — where the button actually is:
| Before | After | |
|---|---|---|
| Desktop CTA position | (440, 303) |
(400, 732) — moved 428px |
| Desktop CTA size | 400×46 | 480×56 |
| Mobile CTA position | (25, 292) |
(24, 771) — moved 479px, now pinned to viewport bottom |
| Mobile CTA size | 400×46 | 364×56 |
Every check passed. The button that returning users tap without looking is now somewhere else, a different size, and — on mobile — permanently fixed to the bottom of the viewport, a known collision point with on-screen keyboards on some mobile browsers, at precisely the moment a user finishes typing their email and reaches for it.
Full diff, full test output, full review notes, and the reproduction steps are in the public repo.
What UXSense already knows about this exact element — real, not simulated
We didn't run a live drift check against our demo (we don't have the ability to instrument it against real traffic in this window, and we're not going to fake one). Instead we pulled UXSense's real, already-recorded behavioral data for the actual production element this change targets:
- The real "Continue to payment" button on
/passenger: 28 real sessions recorded against it, a stereotypy score of 0.95 — UXSense's measure of how ritualized returning-user interaction with an element is. High stereotypy means people interact with it the same way, session after session, without hesitation. - The real passenger-form container: 33 sessions, stereotypy 0.64 — more variable, consistent with a form users are actually reading rather than skimming past.
We also asked UXSense's release brief, grounded only in this recorded data, what an engineer should know before touching this exact page. Its answer, unedited:
"A redesign risks masking this if it moves or renames that button without first understanding why people aren't reaching it... A redesign that changes the form structure or layout could shift that behavior in ways that are hard to attribute without a clear pre-ship baseline."
That's the layer none of the four conventional checks above operate at. A 0.95 stereotypy score is a direct, numeric statement that people click this button from memory, not by reading it — which is exactly the behavior a 428–479px relocation and a resize puts at risk. Tests, types, review, and pixel-diffs have no representation of that number at all.
We're not going to pretend everything caught fire
Two things we checked came back clean, and we're saying so rather than only reporting the scary parts:
- Keyboard tab order was unchanged in this specific diff — we measured it directly (8 tabs from page load, recorded focus target at each step). The button's position in the DOM relative to its siblings didn't change here, so this particular risk didn't materialize in this diff. It's a real risk class for this type of change in general (reordering fields "for a cleaner single-column read" is a very plausible follow-up edit that would trigger it) — just not one this specific commit tripped.
- The mobile visual-regression pixel-diff genuinely couldn't run (the before/after screenshots came out at different pixel dimensions once the layout collapsed to a shorter page) — so we reported the measured button geometry instead of fabricating a percentage.
The point
None of this means AI coding agents are unsafe, or that this specific PR should have been blocked. The change does exactly what was asked, correctly. The point is narrower and more useful: "tests pass, review passed, visual diff passed" is not the same statement as "this still works for the person who already knows how to use it." That gap is invisible to every check most teams run today, and it's exactly where a well-executed, correctly-reviewed AI-authored change can quietly cost you the users who were closest to converting.
Try it yourself
- Full repo, diff, and raw check output: [link — landing page]
- If you want to check whether a change you're about to ship touches an element with real recorded behavioral weight, [try it on your own PR].
- We'd genuinely like to hear it if this has happened to you — a change that passed everything and still broke something real. Reply or comment; it's the next case study.