📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems have achieved near-complete automation of core engineering tasks in AI research, with benchmarks indicating saturation. The residual challenge remains in automating research itself, which may be inherently more complex.
Recent benchmarks indicate that AI systems can now automate the majority of core engineering tasks involved in AI research, with capabilities reaching near saturation levels across multiple metrics. This development suggests that engineering work in AI is effectively automated, shifting the remaining challenge to automating the research process itself.
Six key benchmarks measuring AI proficiency in AI R&D tasks show rapid progress, with three reaching or approaching saturation within 16 to 21 months. For example, the CORE-Bench, which tests research reproduction, improved from 21.5% in September 2024 to 95.5% in December 2025, with the benchmark’s author declaring it ‘solved.’ Similarly, the MLE-Bench, evaluating Kaggle competition performance, rose from 16.9% in October 2024 to 64.4% in February 2026, approaching mid-tier human performance.
These benchmarks assess tasks such as reproducing research, optimizing models, and designing kernels, all of which are core engineering activities. The pattern across these measures suggests that AI has reached a point where engineering tasks in AI development are largely automated, reducing the need for human intervention in these areas.
However, Clark’s analysis leaves open whether the remaining challenge—automating the research process itself—is fundamentally different or more complex. The question remains whether research activities, which may involve creative insight and hypothesis generation, can be automated at scale or if they will continue to require human ingenuity.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
AI Engineering: Building Applications with Foundation Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
Ultimate CI/CD for Platform Engineering: Master DevOps Pipelines, GitOps, DevSecOps, Infrastructure as Code, Multi-Cloud Deployment, and AI-Driven Delivery Automation (English Edition)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
AI research reproduction benchmarks
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications for AI R&D and Innovation Pace
This rapid automation of engineering tasks in AI research could accelerate the development cycle, reduce costs, and shift the competitive landscape. If research itself becomes automatable, the traditional bottlenecks in scientific discovery may diminish, leading to faster innovation cycles. Conversely, if research remains resistant to automation, the residual challenge may limit the overall impact of AI on scientific progress.
Progress of AI in Core Engineering Tasks
Recent empirical work, notably Thorsten Meyer’s analysis of Jack Clark’s recent essay, highlights that AI has made significant strides in automating core engineering skills necessary for AI R&D. The benchmarks—CORE-Bench, MLE-Bench, and kernel design—show consistent patterns of rapid improvement and approaching saturation. These developments build on prior trends indicating that AI is increasingly capable of handling complex, multi-step engineering tasks that previously required human expertise.
This progress aligns with broader industry observations that AI models are moving from experimental tools to production-grade systems capable of automating routine yet critical engineering activities, such as reproducing research, optimizing models, and designing hardware kernels.
“The pattern across multiple benchmarks indicates that AI can today automate vast swaths, perhaps the entirety, of AI engineering.”
— Thorsten Meyer
Remaining Challenges in Automating AI Research
While engineering tasks appear largely automated, it remains unclear whether the research activities—such as hypothesis formulation, creative insight, and experimental design—can be fully automated at scale. Clark leaves this as an open question, suggesting that research may involve dimensions that are inherently less amenable to automation, or that the residual challenge may close faster than anticipated if research itself becomes a form of engineering at scale.
Future Developments in AI-Driven R&D Automation
Over the next 32 months, industry and academia will likely focus on advancing the automation of research activities, testing whether creative and hypothesis-driven aspects can be integrated into AI systems. Monitoring progress on new benchmarks, exploring hybrid human-AI research models, and developing tools for automating hypothesis generation will be key. Additionally, the industry will reassess institutional strategies, potentially shifting investment away from traditional research to automation-focused development.
Key Questions
What are the main benchmarks indicating AI’s progress in automation?
Key benchmarks include CORE-Bench (research reproduction), MLE-Bench (Kaggle competition performance), and kernel design tasks. These measure AI’s ability to perform core engineering activities in AI research and development, with recent results showing rapid improvement and approaching saturation.
Does this mean AI can now fully replace human AI researchers?
Not yet. While engineering tasks are increasingly automated, the research process—particularly creative aspects like hypothesis formulation—remains less certain. It is still an open question whether AI can fully automate research activities at scale.
What implications does this have for the future of AI development?
If engineering is automated, the pace of AI development could accelerate significantly. The remaining challenge is automating research itself, which could lead to faster innovation cycles or, conversely, reveal inherent limits to automation in scientific discovery.
Are there risks associated with automating AI research?
Potential risks include over-reliance on automated systems, reduced human oversight, and the possibility of missing creative or ethical considerations that are harder to encode in AI systems. These concerns will need ongoing monitoring as automation advances.
Source: ThorstenMeyerAI.com
