Is AI Coding Better Than Human Coding?

A cross-study analysis of 31 sources from 2025 and 2026: maintenance, legacy system modernization, and developer productivity

AI can produce code faster. That does not yet mean it can produce better software faster.

The question in the title actually contains four different questions:

1. Quality and Durability - is the resulting code good, and does it survive in the project?

2. Speed and Output - does the developer get more work done, or get it done faster?

3. Build Success - can the generated or migrated system even be built and run?

4. Behavioral Equivalence - does the new system behave the same as the original?

And the research shows an uncomfortable thing: these four indicators often move in opposite directions. AI increases the number of pull requests but makes review longer. Migrated code compiles but fails the tests. The new application looks like the original but does not preserve its business rules.

I divided the evidence into three groups: maintenance of existing code, modernization of legacy systems, and developer productivity. Only sources published or peer-reviewed in 2025 and 2026 are used. Given how dynamically this field is evolving, older studies seemed irrelevant to me. The complete studies and figures can be found in the tables and in the references. The text between them does not try to retell every study individually. It tries to answer three practical questions: what direction the evidence points in as a whole, where the biggest risk lies, and what a manager should do differently.

And the header should probably be: Is Coding with AI Better than Coding without AI?

How to Read the Tables

I rate each criterion on a color scale:

🟦 AI wins strongly (difference in multiples) · 🔵 AI wins mildly (difference in percentage points) · ⬜ draw (measured, difference insignificant) · 🟡 human or non-AI wins mildly · 🟨 human or non-AI wins strongly · - not measured

For each study I also list the type of evidence, because not every piece of evidence carries the same weight. In descending order: randomized experiment (RCT), meta-analysis, peer-reviewed empirical study or benchmark, industry case study, telemetry or survey without a control group, blog.

And one confession right at the start: the number of publications is not the number of independent pieces of evidence. Several maintenance studies analyze the same dataset of agentic pull requests, so five papers can be five analyses, but not five independent experiments.

And one more confession: while creating this analysis, I worked with Claude Fable and ChatGPT 5.6.

1. Maintenance: AI in Maintaining Existing Code

Maintenance studies track pull requests: merge rate, time to integration, churn, and how long the code survives in the project.

Across the maintenance studies, the picture is surprisingly uniform. In speed and volume, AI wins: agentic changes are created faster and there are more of them. In quality, it did not win once - of the four direct human-versus-agent quality comparisons, two favored the human and two ended mixed.

The strongest signal, however, is not about defect rates but durability. AI code had 79% higher churn and survived roughly 11 times shorter in the project (a median of 3 days versus 34 days). The agent does not necessarily write buggier code. It writes code that does not take root in the project. Add to that weaker discipline: in performance optimizations, humans attached real benchmark results to nearly half of their PRs, agents to only a quarter.

• Direction of the evidence: AI creates changes faster and in higher volume.

• Main risk: quality and durability do not improve automatically - the evidence points rather in the opposite direction.

• Management response: strengthen review and testing, and track rework and churn, not just the number of changes.

Management takeaway: maintenance

AI can increase both the speed and the volume of code changes, but the coding time saved may come back as review and rewriting. Deploy it where a change can be checked quickly, and measure the whole delivery process, not just code output.

2. Legacy System Modernization: from Generation to Controlled Transformation

Modernisation is a harder discipline than maintenance. Producing valid code is not enough - the new system must preserve business rules, data transformations, user workflows, edge cases, and integration contracts.

One thing does not belong in the table because nobody measured it: not one of the ten modernization studies tracked long-term maintainability after deployment. They measure compilation, tests, and visual similarity, not what the system will cost three years from now.

Three consistent patterns emerge from the ten modernization studies.

First: direct prompting is not a strategy. Wherever a study compared plain prompting with a structured approach, the structured approach won - in the most extreme case, 0% compilable outputs versus 90.67% on the same task. It is not the model that creates success. It is the system around the model: dependency analysis, API mapping, a compiler, tests, a repair loop, and human review.

Second: build success lies. The median build success rate across the quantified migrations was 52.9%, while the median behavioral equivalence was only 33.3%. Where a single study reports both figures at once, the gap is brutal: AlphaTrans achieved 96.4% syntactic correctness but only 25.14% functional equivalence. Code that compiles is not yet a system that works.

Third: boring deterministic tools are still alive. A rule-based transpiler is a tool that translates code from one language into another according to pre-written rules: no AI, just the mechanical mapping "this construct of the old language always translates into this construct of the new one". Where a transformation is formally definable in this way, a plain rule-based transpiler beat generative models in a direct comparison. And the only case of full behavioral equivalence in the entire table - the translation of scientific code - was achieved thanks to a pipeline in which the original program served as an oracle, that is, as the source of truth: the correctness of every translated piece of code was verified by comparing its output with the output of the original. Even so, it cost 10 to 15 hours of human supervision.

• Direction of the evidence: AI significantly speeds up transformation work.

• Main risk: build success is substantially higher than behavioral equivalence.

• Management response: use AI inside a controlled, incremental verification pipeline.

Management takeaway: modernization

AI is useful for transforming parts of a legacy system, but a successful build is not proof of a successful modernization. Require tests, behavioral comparison with the original system, a repair loop, and incremental deployment.

3. Productivity: the Developer, the Team, and the Whole Company

Pull-request studies measure finished output. Productivity experiments ask a different question: would the same person have been better without AI?

At first glance, the results contradict each other: from a 19% slowdown to a 26% speedup. In reality, it is not a contradiction. It is context. The speedup was measured by large field experiments on bounded, well-defined tasks. The slowdown was measured by the METR experiment on senior developers working in code they had known intimately for years - and who, even after the experiment, remained convinced that AI had sped them up by 20%. The feeling of speed and actual speed are two different things, and our brains root for AI. The best combined estimate comes from a meta-analysis of 23 studies: a mild positive effect (g = 0.33), larger in laboratory tasks, smaller in enterprise and open-source settings.

The second pattern: more output does not mean a faster company. Teams with high adoption show almost twice as many merged PRs - and, at the same time, 91% longer review times and considerably larger PRs. The bottleneck did not disappear; it moved from writing to review. Telemetry sources add growing code duplication and declining refactoring; this is directional evidence without a control group, but consistent with the maintenance block above.

The third pattern: AI does not help everyone equally. It raises juniors’ output more than seniors’. At the same time, however, a controlled experiment showed 17% worse understanding of the technology being learned when working with AI. AI helps a junior finish the task. That does not yet mean it helps them become a senior.

• Direction of the evidence: AI often improves output, but the result depends heavily on context.

• Main risk: the bottleneck moves into review, debugging, and integration.

• Management response: deploy AI by task type, not as one blanket policy for the whole team.

Management takeaway: productivity

AI is not one identical tool for every developer and every task. It is most valuable where context is limited, the task is clear, and the output is easy to verify. In deeply familiar code, experienced developers may sometimes be faster without it.

4. Where Should AI Be Used in Modernization?

The main management question is no longer whether to use AI everywhere. It is: where does AI have the conditions to succeed? Four practical rules follow from the evidence above.

Deploy aggressively where the result is easy to verify. Repetitive code transformations, boilerplate, test generation, documentation, clearly specified API changes, isolated migration steps - everything checked by a compiler, tests, or deterministic rules. Here, AI reduces manual work at a manageable risk.

Proceed carefully where business behavior is only partly documented. Legacy workflows, hidden business rules, data-dependent behavior, user-facing processes, integrations built over many years, code that only a few experienced people still understand. Here, AI can generate convincing-looking code that does not preserve the behavior the business actually runs on.

Do not measure success by generated code. The number of generated lines, the number of pull requests, the percentage of code written by AI, or initial build success alone are not success metrics. Measure operational indicators: review time, test success, rework, rejected changes, churn, production incidents, behavioral equivalence, and the time until a change is safely deployed.

Modernise incrementally. The original application stays in operation and serves as the behavioral reference, the system changes in controlled steps, every migrated component is validated, AI is combined with deterministic tooling, and old and new components coexist. One big AI rewrite is not the default strategy - it is the riskiest option in the entire table below.

The table is not a universal formula. It is a practical starting point for deciding where to let AI in first and what control to build around it.

5. Summary: Cross-Study Scorecard

The counts are at the level of studies, not fully independent experiments (the agentic PR dataset repeats within the maintenance block).

The Most Important Findings in Numbers

If you remember only five numbers from this entire analysis, let them be these:

26% faster coding. The strongest controlled evidence in AI’s favor: three field experiments with 4,867 developers measured +26% completed tasks.

0% higher code quality. Of the eight direct quality comparisons, AI did not win a single one.

37.5% lower code quality. In three of the eight quality comparisons (37.5%), human code was clearly better; the rest ended in a draw or mixed.

33% behavioral equivalence. The median of four repository-level migrations. The bar for a finished system is, by definition, 100% - and you have to iterate your way there through tests, an oracle, and a human.

53% build success. The median of five quantified migrations. Raw AI output therefore covers a third to a half of the way to a finished modernisation; the verification system delivers the rest.

These are practical cross-study indicators built on different research methods and definitions. Software development cannot be reduced to one universal metric, but these figures show the clearest direction of the evidence currently available.

Translated into one sentence: AI currently has its clearest advantage in producing output. Its weakest evidence is in long-term quality and in preserving the complete behavior of modernized systems.

Conclusion: Where to Trust AI and Where to Control It

The question is no longer whether companies should use AI in software development. Most already do - according to the 2025 Stack Overflow survey, 84% of developers, even though trust in the accuracy of its outputs fell year over year from 40% to 29%.

The real management question is: where to trust AI, where to constrain it, and what verification system must surround it.

In modernization, it pays to deploy AI most aggressively on tasks that are repetitive, clearly defined, and easy to test. More carefully where the real specification exists only in the behavior of the legacy application. The DORA 2025 conclusion fits here: AI is an amplifier - it magnifies the strengths of disciplined teams and the chaos of everyone else.

AI can accelerate modernization. It cannot decide what must be preserved.

AI does not eliminate the work required to modernize software. It reduces part of the manual rewriting and shifts the center of effort toward reconstructing context, verification, review, testing, and integration.

Three Questions for Discussion

1.       Which parts of your modernization process are defined and testable enough to be delegated to AI?

2.       Where has AI genuinely removed work for you - and where has it merely moved it into review and fixes?

3.       How does your organization prove that the modernized system behaves the same as the original one?

I did this cross-study analysis at the end of July 2026. AI evolves in leaps, and many of these conclusions will probably no longer hold in a few months. I keep being excited.

References

A. Maintenance, Agentic Pull Requests, and Code Review

1.       On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub. TOSEM 2026.

2.       How AI Coding Agents Modify Code: A Large-Scale Study of GitHub Pull Requests. MSR 2026.

3.       Code Change Characteristics and Description Alignment: A Comparative Study of Agentic versus Human Pull Requests. MSR 2026.

4.       Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests. MSR 2026.

5.       Behind Agentic Pull Requests: An Empirical Study on Developer Interventions in AI Agent-Authored Pull Requests. MSR 2026.

6.       How Do AI Coding Agents Contribute to Software Development? An Empirical Study of Agentic Pull Requests. 2026.

7.       When Code Authors Are Agents: A Large-Scale Study of Human-Agent Collaboration in Pull Requests. 2026.

8.       Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories. 2026.

9.       3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse. 2026.

10.   How Do Agents Perform Code Optimization? An Empirical Study. MSR 2026.

B. Legacy System Modernization and Code Migration

1.       Porting Declarative UI to HarmonyOS: A Heuristic-Guided LLM Approach (ArkTrans). 2026.

2.       GUIMigrator: Semantics-Preserving Transpilation from Android XML to Compose and SwiftUI. FSE 2026.

3.       LLM-Powered Migration of UI Component Libraries. Zalando Engineering, 2025.

4.       LegacyTranslate: LLM-Based Multi-Agent Method for Legacy Code Translation. 2026.

5.       Migrating Code at Scale with LLMs at Google. FSE 2025.

6.       How Is Google Using AI for Internal Code Migrations? 2025.

7.       RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation. TSE 2026.

8.       RustRepoTrans: Repository-Level Context Code Translation Benchmark Targeting Rust. ASE 2025.

9.       AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation. FSE 2025.

10.   Systematic LLM Translation of Legacy Scientific Code to Differentiable Frameworks. 2026.

C. Productivity, Maintainability, and Skill Growth

1.       The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science 2026.

2.       Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, 2025.

3.       We Are Changing Our Developer Productivity Experiment Design. METR, 2026.

4.       How AI Impacts Skill Formation. 2026.

5.       Echoes of AI: Investigating the Downstream Effects of AI Assistants on Software Maintainability. Empirical Software Engineering, 2026.

6.       A Meta-Analysis of the Effect of Generative AI on Productivity and Learning in Programming. 2026.

7.       The AI Productivity Paradox. Faros AI, 2025.

8.       State of AI-Assisted Software Development. DORA / Google, 2025.

9.       AI Copilot Code Quality. GitClear, 2025.

10.   The Maintainability Gap. GitClear and GitKraken, 2026.

11.   Stack Overflow Developer Survey 2025. Stack Overflow, 2025.