TL;DR
A randomized trial at Boston Consulting Group found consultants using ChatGPT scored higher than a group using Google and reference sites on coding, statistics and prediction tasks. After the chatbot was removed, researchers found no evidence that the ChatGPT group performed better on a technical knowledge test.
A randomized trial involving Boston Consulting Group consultants found that those given ChatGPT and brief prompting training scored higher on data science tasks than colleagues using Google and reference sites, but showed no measurable advantage on a later test without the chatbot. The findings, published in Nature Human Behaviour on October 26, 2026, suggest that the performance gains in this trial depended on access to the tool.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The trial took place at BCG in late March and early April 2024. Of 986 consultants allocated to the study, 573 began the survey and 487 submitted work on both assigned tasks; that 487-person group formed the main analysis sample. Participants had at least a bachelor’s degree, and the authors report that most said they had no coding experience.
The treatment group received 15 to 20 minutes of ChatGPT prompting instruction and used GPT-4. The control group received instruction on using Google and sites including Stack Overflow and Khan Academy. Each participant completed two 90-minute tasks drawn from coding, statistics and prediction. A separate group of 44 BCG data scientists completed the tasks without AI, providing an expert benchmark.
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The study reports that participants still scored below the data scientists on average. After the tasks, participants answered five technical questions without ChatGPT. The researchers found no evidence of an advantage for the ChatGPT group: the reported differences ranged from −7.7 to +4.8 percentage points, and the joint test was not statistically significant (p = 0.531).
Performance Gains With ChatGPT
The results distinguish between doing a task with AI support and retaining the ability to do similar work independently. In this trial, short training and access to ChatGPT helped consultants complete tasks beyond their reported technical experience, while the follow-up test did not show a measurable knowledge advantage once the tool was removed.
That distinction matters for employers evaluating AI as a form of workforce training. Task performance while using a chatbot may show that employees can produce more with the tool; it does not by itself establish that they have learned the underlying skills. The study’s findings are limited to its participants, tasks and testing setup, but they give companies a reason to measure unaided skills separately from AI-assisted output.
The authors also report an exploratory result: participants in the ChatGPT group became more confident that the system could solve problems it could not actually solve. Because the authors describe this analysis as exploratory, it should be treated as a finding for further study, rather than a settled conclusion about AI users generally.
Products Worth Considering
Apple Watch SE 3 GPS 44mm Smartwatch
DJI Osmo Mobile 8 Phone Gimbal with Mic Mini Transmitter, Infinity Black
Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds with Immersive Audio and Wireless Charging Case, Black
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Worked
The researchers preregistered the trial on March 13, 2024, before collecting data. Consultants completed two of three 90-minute exercises: a Python task involving cleaning and combining datasets; a statistics task involving model errors, residual plots and probability; or a prediction task about the predictability of international football matches.
According to the source article’s account of the paper, BCG data scientists developed the tasks with OpenAI’s Economic Impacts Research team. The tasks were designed so ChatGPT could offer useful guidance but could not independently answer every question correctly. This setup was intended to test participants’ work with the tool, rather than whether they could simply copy a complete correct response.
The study, titled “Generative AI and the temporary upskilling of knowledge workers,” describes AI as an “exoskeleton”: a technology that can extend what a person can do while in use. The authors say the observed task gains were temporary in the sense that the researchers found no evidence of a corresponding gain on the test taken after ChatGPT was removed.
““find no evidence” that the ChatGPT group did better on the technical questions answered without the tool.”
— The study’s authors
Limits of the Follow-Up Test
The follow-up assessment consisted of five technical questions. The reported result means the study detected no measurable advantage for the ChatGPT group on that test; it does not establish that participants learned nothing, or settle whether different training, longer practice or another assessment would produce a different result.
The source material does not provide enough detail to assess how well the results apply to other employers, occupations or current AI systems. The trial used GPT-4 in 2024 and selected tasks developed for the experiment. It also does not establish whether longer-term use of AI can build skills, or how the reported confidence effect would hold up in a larger study.
Testing Skills Beyond AI Access
The paper’s findings leave employers with a practical measurement question: whether staff can carry out comparable work after AI support is withdrawn. Companies using chatbots for technical tasks can track AI-assisted output and unaided performance separately, while checking work for errors during use. The study does not prescribe a specific workplace training method.
Further research would be needed to determine whether longer or differently designed instruction helps workers retain skills, and whether the results extend beyond this trial’s consultants and tasks. The source material does not identify a follow-up study or a scheduled next milestone.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than the Google and reference-site control group on coding, statistics and prediction tasks. On a five-question test taken without ChatGPT, researchers found no measurable advantage for the ChatGPT group.
How much higher were the ChatGPT group’s task scores?
Compared with the control group, the ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction.
Did the ChatGPT group match BCG data scientists?
No. The study reports that the consultants’ average scores remained below those of the 44 BCG data scientists who completed the tasks without AI.
Does the study prove ChatGPT cannot teach data science?
No. It tested a specific training session, tool, set of tasks and short follow-up assessment. The authors found no evidence of a knowledge advantage on that test, but the results do not settle what different or longer training might achieve.
Source: Thorsten Meyer AI
