COBOL migration has spent decades being treated as one of those enterprise problems that is too expensive, too risky, and too deeply embedded to solve cleanly. That assumption is getting harder to defend.
OpenAI's latest coding results suggest that AI can now translate significant portions of COBOL with far greater accuracy than earlier models. In its October 19 evaluation, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol. At higher reasoning effort, OpenAI reports 97.9%. The company also says the newer model achieved those low-effort results at 61% of the previous model's API cost.
Those numbers are large enough to change the conversation around legacy software.
They do not, however, prove that enterprises can hand a decades-old COBOL estate to an AI model and receive a modern application in return.
That distinction matters because code translation and system modernization are different engineering problems.
OpenAI's benchmark measures whether a program can be translated while preserving observable behavior under a hidden test suite. Real modernization requires understanding the surrounding system, its data, integrations, operational processes, undocumented business rules, deployment architecture, and the parts of the application nobody fully understands anymore.
AI is making the first problem much cheaper. The second problem remains an architecture and verification challenge.
What OpenAI Actually Tested
OpenAI published two related evaluations for low-resource programming languages such as COBOL, Fortran, Visual Basic and OCaml.
The first is MultiLangBench, a public benchmark that asks models to build a SQLite-compatible database engine from scratch in 20 programming languages. The resulting implementations are tested against a reference SQLite implementation using tests adapted from SQLite's existing test suite.
At maximum reasoning effort, GPT-6 Astra achieved an average 87.4% test pass rate across the 20 languages, compared with 26.8% for GPT-5.6 Sol. COBOL improved from 10.9% to 72.1%, although it remained the hardest language in the benchmark. OpenAI has released the MultiLangBench setup and test suite, making this part of the announcement independently inspectable.
The second evaluation is more relevant to migration.
OpenAI asked models to translate COBOL programs into C++, Java, Kotlin, Python or Rust while preserving their behavior. The evaluation uses hidden tests and requires byte-for-byte parity on observable outputs. A migration counts as successful only when every test passes.
GPT-6.1 Sol reached 86.6% at low reasoning effort and 97.9% at higher reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same low-effort setting.
That is a significant improvement.
There is also a limitation that enterprise decision-makers should put directly beside those numbers: the migration evaluation itself is not public.
OpenAI has not released the underlying COBOL programs or hidden tests, so outside teams cannot independently reproduce the 86.6% or 97.9% figures. The New Stack and other analysts have called attention to that difference between the public MultiLangBench and the private migration evaluation.
That doesn't make the results meaningless.
It means they should be treated as evidence of progress, not as an independently verified estimate of how accurately an AI system will migrate an enterprise mainframe.
The Benchmark is testing a narrower problem than modernization
A successful translation has a clear definition in OpenAI's evaluation: the new program produces the same observable outputs as the original program under the hidden test cases.
That's a good standard.
It is also much narrower than replacing a production system.
A real COBOL application may sit inside a much larger environment containing copybooks, batch jobs, databases, message queues, schedulers, authentication systems, reporting systems, downstream applications, operational procedures, and integrations with software written in several other languages.
IBM makes this distinction in its own modernization guidance, describing COBOL modernization as something that involves the application, the platform where it runs, and the surrounding technology stack. Migration can involve moving workloads to cloud platforms, changing data access, introducing middleware, and modifying how the application interacts with its environment.
The code is only one part of the system.
That is why a model can become exceptionally good at translating a COBOL program while an enterprise migration remains difficult.
The translation question is:
Can we produce another implementation that behaves like this program?
The modernization question is:
Can we replace this part of the business system without breaking everything that depends on it?
Those questions overlap, but they aren't interchangeable.
COBOL's business logic is often hidden in behavior
OpenAI's own example illustrates why this matters.
COBOL's fixed-point arithmetic can discard excess decimal digits unless rounding is explicitly requested. A migration that changes that behavior can still compile, execute, and appear correct while producing different financial results. OpenAI specifically uses this example to explain why its hidden tests require behavioral parity.
Imagine a banking calculation that historically produces a particular amount because of an old rounding rule embedded in the implementation.
A developer might look at the rewritten code and consider it cleaner. The application starts normally. The API responds. The database updates.
The result can still be wrong.
That is the central risk in legacy migration: the behavior that matters may not be documented anywhere except in the way the existing system behaves.
Business rules can be buried in arithmetic, record layouts, condition handling, batch sequencing, exception paths, and decades-old workarounds.
AI can read those patterns.
It still needs evidence that its interpretation matches reality.
The missing piece is system decomposition
This is where the criticism of OpenAI's benchmark becomes important.
Software evolution researchers have pointed out that the difficult part of large-scale legacy modernization is often breaking a complex system into smaller units that can be replaced independently. The Verge's reporting on the benchmark quoted researchers making exactly this distinction: rewriting code is different from replacing a system, particularly when the system spans multiple languages, databases, integrations, and organizational processes.
That is a problem no code-translation benchmark can fully capture.
Consider a hypothetical claims-processing platform with several million lines of COBOL. One program calculates eligibility. Another reads records from a mainframe database. A batch job triggers processing overnight. Several downstream systems consume the generated records. A reporting application depends on particular field formats. Operations teams have decades of procedures built around the batch schedule.
Translating each COBOL program into Java does not automatically produce a modern claims platform.
You still have to decide which component should move first, which interfaces should remain stable, which data should move, which behavior must remain identical, and which parts of the system should actually change.
That is architecture work.
AI can help with it, but translation alone doesn't solve it.
The right question is no longer"Can AI Translate COBOL?"
That question is becoming less interesting.
The better question is:
Which parts of a COBOL modernization program can now be automated safely, and what verification is required around them?
AI can already contribute to several stages of the process.
It can explain unfamiliar programs, identify data flows, generate documentation, translate individual modules, produce tests, compare implementations, suggest refactoring, and help engineers understand dependencies. IBM's modernization work describes similar uses for AI, including code understanding, conversion, refactoring, test generation, and documentation.
Research from IBM also shows why translation increasingly works best when combined with program analysis rather than relying on an LLM alone. Its enterprise COBOL-to-Java work combines static analysis with LLM-based translation to infer classes, hierarchies, method signatures, and program structure before generating Java code.
That hybrid approach is revealing.
The useful architecture isn't necessarily:
COBOL → LLM → Java
It is closer to:
COBOL → program analysis → dependency and behavior model → AI transformation → automated verification → engineer review
The AI becomes one component in a migration pipeline.
Verification becomes the new bottleneck
There is an economic consequence to better translation.
If AI makes translation dramatically cheaper, organizations may spend less money generating replacement code and more money proving that the replacement behaves correctly.
That isn't a bad trade.
It is probably the correct one.
IBM researchers have already published work on automated testing for COBOL-to-Java transformation, using techniques including symbolic execution to generate tests and compare the behavior of original COBOL programs against translated Java implementations. Their paper describes manual validation of translated code as necessary but time-consuming, which is why automated equivalence testing becomes valuable.
Another 2026 benchmark, LegacyCodeBench, takes a similar verification-first approach to evaluating AI comprehension of COBOL. Rather than asking an LLM to judge whether its own explanation is correct, it extracts behavioral claims and verifies them by executing the original program.
These approaches point toward an important change in modernization economics.
The scarce resource may shift from writing replacement code to proving that the replacement is correct.
That should change how migration projects are budgeted.
A Safer AI-assisted migration starts with characterization
Before asking an AI system to rewrite a critical COBOL workload, teams need to establish what the existing system actually does.
This is sometimes called characterization testing: capture the behavior of the current system before changing its implementation.
The goal isn't to prove that the existing code is correct.
The goal is to establish a behavioral reference.
For important workloads, that means collecting representative inputs and outputs, testing edge cases, documenting external dependencies, identifying data transformations, and capturing the operational conditions under which the application runs.
Financial systems need particular attention to arithmetic and rounding. Batch systems need timing and sequencing tests. Record-processing systems need malformed-input and missing-record cases. Integration-heavy applications need contract tests for the systems around them.
Only after that baseline exists does AI-generated translation become something an engineering team can evaluate systematically.
The benchmark then becomes your own.
Don't start with the whole mainframe
Large modernization programs become dangerous when they treat the mainframe as one application.
A better strategy is to identify a bounded workload with clear inputs, outputs, dependencies, and ownership.
The candidate should ideally have enough business value to justify the work but enough isolation that the team can run old and new implementations side by side.
That gives the organization a controlled migration loop:
Understand → characterize → translate → test → compare → deploy in parallel → observe → cut over
The AI can participate heavily in the middle of that loop.
It can analyze the code, generate the target implementation, write tests, investigate mismatches, and propose corrections. Engineers still own the decision about whether the new implementation is sufficiently equivalent.
That makes the first migration less about proving that AI can rewrite COBOL and more about proving that the organization can verify AI-generated changes.
Run the old and new systems together
Parallel execution is particularly valuable for high-risk workloads.
For a period of time, the existing COBOL implementation remains authoritative while the replacement system processes the same inputs. The outputs are compared automatically.
A mismatch becomes an engineering investigation.
Sometimes the new code is wrong. Sometimes the old system contains undocumented behavior that nobody realized was business-critical. Sometimes both systems are correct but represent data differently.
That feedback is extremely valuable because it turns hidden business logic into observable evidence.
AI can help classify and explain those differences, but the comparison framework should remain deterministic.
The question should not be whether the model thinks two implementations are equivalent.
The question should be whether the systems actually produce equivalent results for the cases that matter.
What the 86.6% number shouldmean to a CIO
A CIO shouldn't interpret OpenAI's result as "97.9% of our COBOL can now be migrated."
That isn't what the benchmark measures.
The useful interpretation is that AI-assisted translation has crossed an important capability threshold. Tasks that previously required large amounts of manual transformation may now be automated or accelerated enough to justify serious pilots.
That changes the business case.
The cost of understanding and translating legacy code can fall. Engineers can spend more time on architecture, verification, integration, and risk management instead of manually rewriting syntax.
But the migration budget doesn't disappear.
It moves.
Some of the money previously allocated to translation may need to move toward automated testing, parallel execution, dependency mapping, data validation, observability, and expert review.
That is a healthier modernization model because those activities are necessary regardless of whether AI performs the translation.
What the benchmark does not tell you
Several important questions remain unanswered by OpenAI's published migration evaluation.
We don't know the complete size and composition of the hidden test suite. We don't know how representative the COBOL programs are of the most difficult enterprise systems. We don't have an independent reproduction of the 86.6% and 97.9% scores. And the benchmark doesn't evaluate the organizational and architectural work required to replace a large production system.
The New Stack and independent commentary have raised these limitations, while OpenAI itself explicitly says production application of the capability still requires expert review and testing against business rules and dependencies.
Those limitations don't invalidate the result.
They define its scope.
MultiLangBench gives the community something reproducible. The COBOL migration evaluation gives us an important but privately measured signal about model capability.
Enterprise teams should use both appropriately.
A practical pilot for enterprise COBOL
A serious pilot doesn't need to begin with millions of lines of production code.
Pick one bounded application or component where the business owner can clearly define what correct behavior means. Build a complete test harness around the existing implementation before generating replacement code. Capture representative production behavior, including edge cases that experienced COBOL engineers know are historically important.
Then have the AI produce a candidate translation.
Don't evaluate it with a code review alone.
Compile and execute it. Compare outputs against the reference implementation. Run negative tests. Exercise boundary conditions. Compare database mutations. Check performance. Inspect integrations. Record every mismatch and determine whether it represents a translation error or undocumented behavior in the original system.
A useful pilot should measure at least four things: translation accuracy, verification effort, defect discovery, and engineering time saved.
The fourth metric is easy to overlook.
If AI generates replacement code in hours but engineers spend months understanding and correcting it, the migration economics look very different from a case where automated verification catches most discrepancies quickly.
What we see at 0xMetaLabs
The COBOL story is becoming less about whether legacy languages can be understood by AI and more about how enterprises should redesign the modernization process around increasingly capable automation.
That is a meaningful change.
For years, organizations often treated legacy modernization as a choice between continuing to maintain the existing system or funding a large rewrite. AI introduces a third possibility: progressively extract, understand, test, and replace parts of the system while keeping the existing platform available as a behavioral reference.
That approach reduces the size of the bet.
It also changes the role of the modernization team. Less effort goes into mechanically translating syntax. More effort goes into identifying boundaries, defining contracts, building verification systems, and deciding which parts of the legacy architecture should survive in the new environment.
AI is well suited to the first half of that work.
Engineering judgment remains essential for the second.
Translation is becoming cheap. Correctness isn't.
OpenAI's COBOL results are significant because they show how quickly AI coding capability is moving into languages that were previously difficult for general-purpose models. GPT-6.1 Sol's reported 86.6% low-effort migration accuracy, rising to 97.9% at higher effort, is a substantial jump from the 5.4% result reported for GPT-5.6 Sol.
The result should encourage enterprises to revisit modernization programs that previously looked economically difficult.
It should not encourage them to skip the engineering work that makes those programs safe.
COBOL modernization is not a translation contest. A successful migration has to preserve business behavior, integrate with surrounding systems, handle real production data, meet operational requirements, and provide enough evidence that the organization can trust the replacement.
AI can make the transformation phase dramatically faster.
The verification phase still needs to be designed.
The opportunity isn't to replace COBOL with AI-generated Java, Python, or Rust. The opportunity is to use AI to make legacy modernization incremental, testable, and economically viable without treating a benchmark score as proof that a production system is ready to switch over.
