Hey everyone, picture a team that treats every coding agent like the same coworker with a different logo. The bake-off deck still leads with accept rates and polished demo repos. The procurement slide says “AI coding” as one line item. Then the merged code sits in production for ninety days, and the story stops matching the slide.
Say you are choosing among Codex, Copilot, Cursor, Claude Code, Devin, or whatever lands in your budget next quarter. The model demos still matter. The part that rarely shows up in the bake-off is coding agent post-merge quality: reverts, security smells in the diff, how hard humans have to review the change, and whether the debt the agent left still exists months later. Two fresh empirical papers give you numbers for that second conversation. One watches agent pull requests for ninety days after merge. The other tracks AI-authored commits until the debt either clears or survives at HEAD. Together they give working engineers a way to talk about agents as tools with measurable aftercare, not as magic throughput switches.
Why “AI code” is the wrong unit of analysis
Imagine a platform lead who would never score every cloud vendor with one uptime number, then scores every coding agent with one “AI productivity” number. That shortcut collapses the differences maintainers actually feel after merge.
Obada Kraishan’s study, Not All Agents Are Equal, follows 37,623 provenance-labeled pull requests across five commercial agents and a matched human baseline in the same repositories (arXiv:2609.17598). The window runs December 2024 through July 2025 across 2,807 GitHub repositories. For merged work, the authors watch ninety days of follow-up. For patch quality, they analyze about 1.35 million added lines in Python, JavaScript, and TypeScript.
The headline for practitioners is blunt. Quality differences are vendor-specific rather than uniform. Codex-authored pull requests were reverted about half as often as human pull requests in the same repos (6.1% versus 11.5%, odds ratio 0.50). Devin pull requests were reverted more often (14.5%, odds ratio 1.31). Copilot, Cursor, and Claude Code sat statistically near the human baseline on reverts. If your dashboard only says “agent PRs,” you average away the decision that actually costs you a fire week.
The security-smell story is also uneven
Pooled across vendors, agent pull requests were less likely than human ones to contain any security smell in the analyzed languages: 2.9% versus 4.6% (odds ratio 0.63). That sounds comforting until you open the per-vendor chart. Cursor and Codex looked cleanest on presence rates. Claude Code sat at 9.5% presence, roughly twice the human rate, while also shipping much larger median diffs. The paper is careful: regex smells on added lines are patterns, not confirmed exploits. Still, the pattern matches what many teams already feel. Some agents leave quieter diffs. Other agents leave noisier diffs that need more human cleanup. Size and task mix almost certainly confound the Claude Code slice, which is why per-line densities and size buckets matter in the paper’s sensitivity checks.
Two weakness classes were under-represented in pooled agent code after correction: hardcoded credentials and eval-style constructs. Even there, vendor spread remains. Credential smells showed a fourfold gap between the cleanest agents and Claude Code’s higher rate. “AI is either safe or unsafe” is the wrong sentence. “Which agent, on which class of change, under which review budget?” is the sentence that survives contact with the data.
Review capacity is already part of post-merge quality
Coding agent post-merge quality is not only about what the static analyzer finds. It is also about how much scarce human attention the change consumes. In Kraishan’s review tables, Copilot pull requests drew the deepest human scrutiny on average, with about 3.6 human reviews and 0.43 change requests per pull request, plus heavy bot review traffic. Claude Code pull requests waited the longest for a first human review, with a median of 12.6 hours against one-to-four hours for other agents. Larger agent diffs almost always wait longer in the human review queue. None of that pattern is mysterious once you watch queue depth. It is ordinary capacity math under a fixed reviewer pool.
Picture a team that celebrates agent merge volume while the same senior reviewers are the only people who can safely approve the large agent diffs. Reported throughput went up on the agent dashboard. The review surface did not grow with it. If your scorecard never records time-to-first-human-review or change-request rate by vendor, you are measuring shipping rate and calling it engineering health.
Debt does not leave when the bot leaves the chat
The second primary source widens the horizon from ninety days after a pull request to the longer life of AI-authored commits. Liu and coauthors study Debt Behind the AI Boom on 302.6 thousand verified AI-authored commits from 6,299 GitHub repositories covering Copilot, Claude, Cursor, Gemini, and Devin (arXiv:2603.28592). They attribute commits with explicit Git metadata, then run static analysis before and after each change so introduced issues can be separated from pre-existing noise. They track those introduced issues forward to the repository HEAD.
They identified 484,366 distinct introduced issues. Code smells dominate at 89.3%, with correctness at 6.0% and security at 4.7%. More than fifteen percent of commits from every studied assistant introduce at least one issue, with tool rates ranging from 17.4% for GitHub Copilot to 29.1% for Gemini. The lifecycle number is the one that should land in staff meetings: 22.7% of tracked AI-introduced issues still survive at the latest repository revision, including issues introduced more than nine months earlier.
Net impact is mixed in a useful way. AI-authored commits fix slightly more code smells than they introduce. On correctness and security, they introduce more than they fix, with security landing around one and a half times as many introductions as fixes. That is exactly the shape you would expect if assistants are strong at repetitive cleanup and weaker at deep contextual judgment. The coding agent remains a powerful delivery tool. The tool is not a senior owner of threat models.
Do not confuse merge green with debt cleared
Picture a release train that treats a green merge as the end of the quality story. Liu’s survival curve argues otherwise. Issues that look minor at review time, especially broad exception handlers, unused bindings, and latent insecure patterns, can still be sitting in HEAD half a year later. The cumulative count of surviving AI-introduced issues in that study climbs into the six figures as adoption scales. That is not a reason to ban assistants. It is a reason to treat AI-touched paths as a first-class debt queue with owners and burn-down targets.
Say you are a tech lead who already runs monthly debt reviews for flaky tests and dependency upgrades. Add an AI-attributed lane to that same meeting. Ask which introduced findings still survive, which security rules are net-negative, and which vendors correlate with the sticky findings in your tree. Keep every debt conversation concrete and dated. “We shipped faster” is incomplete without “and this is the debt that still needs an owner.”
A practical scorecard for the next bake-off
Say you are rewriting the AI coding RFP this month. Keep the acceptance demos in the evaluation packet. Add a post-merge panel that leadership can read without a research paper.
- Revert rate by vendor within ninety days, compared against human baselines in the same repositories when you can get them.
- Security-smell presence and density on added lines, broken out by weakness class rather than one blended score.
- Human review load: time to first human review, human reviews per change, and change-request rate, again by vendor.
- Issue survival for AI-attributed commits: percent of introduced static findings still present at HEAD after thirty, ninety, and one hundred eighty days.
- Net issue balance: issues introduced versus fixed, especially for correctness and security rules you already treat as release blockers.
None of those metrics require you to pretend the papers are randomized trials. Kraishan’s agents are not randomly assigned to tasks. Liu’s dataset covers commits that left explicit AI traces, which is a visible subset rather than every quiet paste from a chat window. Both papers say those limits out loud. For a working engineer, that is still enough to retire the “all agents are equivalent” assumption.
What to change on Monday
Imagine a team that currently has one agent license, one shared prompt pack, and one dashboard titled AI adoption. Split the old single dashboard into two explicit panels. Keep adoption as a leading indicator. Add coding agent post-merge quality as the lagging indicator that decides renewals.
Route large agent diffs to reviewers who actually know the subsystem. Require static analysis gates on AI-attributed commits the same way you already gate human commits. Tag vendor identity in your pull-request metadata so ninety-day revert and churn queries are cheap later. When two vendors look similar in the demo, prefer the one whose merged work reverts less and whose introduced issues clear faster in your repos, not the one with the louder launch video.
AI coding agents are tools in a delivery system. Tools deserve version pins, ownership, and outcome metrics after install. The research now gives you outcome metrics that point past the merge button. Use them before the next renewal cycle turns a vendor logo into permanent debt in your tree.
If this article helped, drop a comment with the first workflow you want to try this week. Support the shenanigans buying me a coffee on Ko-fi, and follow Attune IT on YouTube or me on X.