Hey everyone, picture a team standup where someone flips a dashboard and announces that Agent B is winning because its agentic pull request merge rate sits ten points above Agent A. Leadership treats the chart as settled truth, and procurement starts drafting a bake-off memo. Nobody asks the awkward question sitting under the chart: merged and rejected labels are social outcomes, not report cards for model skill.
Say you are an engineering manager who inherited that scorecard. An agent opens a pull request for an auth ticket that touches SSO edge cases, CONTRIBUTING.md naming rules, and a flaky CI policy your team still argues about. If the change lands, you credit the agent. If reviewers bounce it for a missing migration note or an out-of-band approval gate, you blame the agent. That bookkeeping feels neat until you read what actual Agentic-PR studies keep finding: outcome labels mix agent mistakes, human workflow, and silent review trails that never make it into the metric.
What an agentic pull request merge rate actually measures
In 2026, Peralta and coauthors published an MSR study titled “Why Are Agentic Pull Requests Merged or Rejected?” (arXiv:2605.22534, DOI 10.1145/3793302.3793575). They started with 11,048 closed Agentic-PRs from repositories with at least 500 stars, with an initial closed set of 6,774 merged and 4,274 rejected requests. After removing 1,249 bot-only review cases, they refined the set to 9,799 human-reviewed pull requests, then manually inspected 717 representative cases (353 rejected and 364 merged) to recover decision rationale from interaction artifacts.
The finding is blunt enough to hang above every agent leaderboard: rejection outcomes substantially overstate agent error, and merge outcomes often hide reviewer mediation. Inter-rater reliability was strong for the coding work that produced those claims, with Cohen’s κ ≈ 0.90 for rejection categories and κ = 1.0 for merge interaction patterns. That is careful labeling, not vibes-based commentary.
Rejected does not mean the agent failed
Of the 353 rejected Agentic-PRs in the manual sample, only 126 (35.7%) were clear agentic failures (AF). Another 110 (31.2%) were non-agentic or workflow driven (Non-AF), and 117 (33.1%) landed in unknown because no observable rationale showed up in the artifacts. About one in three rejections lacked a readable reason trail, which is a measurement problem before it becomes a model problem.
Imagine a junior engineer whose SSO edge-case fix is rejected because the ticket still needs a security peer review that never got requested, or because the branch title violated a CONTRIBUTING.md convention nobody pasted into the agent prompt. Calling that an agent fail teaches your org the wrong lesson. The tool may have produced workable code while the surrounding process produced the reject.
Merged does not mean the agent shipped solo
Among the 364 merged Agentic-PRs inspected, 56 (15.4%) needed explicit reviewer involvement: 28 feedback loops where reviewers requested changes and the agent revised (7.7%), plus 28 cases where reviewers applied their own commits before merge (7.7%). Another 288 (79.1%) showed no observed feedback loop, and 20 (5.5%) were unknown. A high merge rate can still be a human-assisted merge rate wearing agent-shaped branding.
The interaction pattern also varies by agent in that sample. Copilot accounted for FL or HI in 29 of 49 of its inspected merged PRs (59.2%). OpenAI Codex showed human involvement as HI in only 1 of 167 merged PRs (0.6%). Cursor showed a feedback loop in 1 of 33 merged PRs (3.0%). Those splits belong to this sample rather than a forever ranking, and they still warn against treating one agentic pull request merge rate as a universal skill score.
Functional tests are not the same as review gates
A second 2026 paper sharpens the same point from the repair bench. SWE-Gate (arXiv:2609.04167) builds 303 repository-level repair instances across 75 open-source Python repositories and looks past green tests. Among 644 repairs that pass functional tests, 221 fail the provided review constraints. Passing the suite is not the same as satisfying the acceptance rules humans actually use in review.
Picture a team celebrating that an agent cleared CI on a permissions refactor while the patch still violates a documented exception policy, omits a required telemetry hook, or ignores a style gate that reviewers always enforce by hand. Your functional dashboard looks green. Your review queue still says no. If you only score agents on tests that already passed, you will overestimate how review-ready the work really is.
What to track instead of a vanity merge scorecard
You do not need a new research program to stop fooling yourself. You need a cleaner scoreboard that treats coding agents as tools inside a human workflow.
- Split rejects into agent failure, workflow or process rejects, and unknown missing rationale, then attack the unknown pile with better review notes.
- Count human mediation on merges: feedback loops, reviewer commits, and silent merges with no interaction trail.
- Pair functional pass rates with review-constraint pass rates, the way SWE-Gate separates those layers.
- Keep agent comparisons inside the same repository norms, because CONTRIBUTING.md rules and CI flake policies are part of the outcome.
- Report an agentic pull request merge rate as a workflow signal beside rework, escaped defects, and time-to-safe-merge, never as a lonely report card.
Say your platform crew is choosing between two coding agents for auth and billing tickets. Run a short bake-off, but score each pull request for why it merged or bounced, who touched the diff, and whether review constraints failed after tests passed. The agent that “wins” on raw merges may be the one that quietly leans on reviewers, or the one that only attempts easy tickets. The agent that “loses” may be the one hitting your sharpest SSO edge cases first.
Treat the merge rate as a dashboard, not a diploma
Coding agents remain tools, and tools inherit the measurement culture around them. If your culture treats every reject as model stupidity and every merge as model genius, you will buy the wrong upgrades and coach the wrong habits. The MSR 2026 evidence says a large share of rejects are not clean agentic failures, and a meaningful share of merges still need humans in the loop. SWE-Gate adds that green functional tests can still fail review constraints more than a third of the time in that repair set (221 of 644).
Keep using coding agents on real tickets, and keep reviewing their diffs with the same seriousness you give a strong junior hire. Just stop grading the model with a binary stamp that was never designed to measure skill alone. Your agentic pull request merge rate is useful when it points you at process friction. It becomes harmful when it becomes the only number in the room.
If this article helped, drop a comment with the first workflow you want to try this week. Support the shenanigans buying me a coffee on Ko-fi, and follow Attune IT on YouTube or me on X.