AI can now propose hypotheses and run experiments. What's still missing?

calendar_today ago • schedule7 min read

On March 25, 2026, Nature published a paper from Sakana AI in Tokyo, the University of British Columbia, the University of Oxford and others describing a system called the AI Scientist. It comes up with research ideas, searches the literature to check that they are new, writes and debugs code, runs experiments, makes plots, analyzes the data, writes a complete paper and then scores its own work. The same paper is candid about the weak spots: some ideas are underdeveloped, citations can be wrong, the system works only on machine learning research for now, and it still needs human oversight.

An "agent" here means a language model that does more than answer questions. It writes a plan, calls tools such as literature search, a code interpreter or lab equipment, reads back what happened, and adjusts its next step, over and over. The AI Scientist runs in two modes: one starts from a code template supplied by humans for a set topic, and the other has no template and searches more widely for directions on its own.

Research agents like this have multiplied over the past three years. It is worth being specific about what they have actually done, and where they keep getting stuck.

What they've done

The first clear sign that a language model could run real experiments was Coscientist, published in Nature in December 2023. Gabe Gomes's group at Carnegie Mellon University used GPT-4 as a planner that could search the web, read instrument documentation, write control code and send instructions to a cloud-based automated lab. Given a set of reagents, it worked out procedures for Suzuki and Sonogashira couplings, two workhorse palladium-catalyzed reactions, and carried them out successfully.

A few months later Nature Machine Intelligence published ChemCrow, from a team at EPFL in Switzerland and collaborators in the United States. It connects GPT-4 to 18 tools designed by chemists, covering tasks such as converting between molecule formats, predicting reactions and flagging hazardous compounds. The authors report that ChemCrow planned and executed the synthesis of an insect repellent and three organocatalysts on its own, and guided the discovery of a new chromophore.

By 2025, agents were working in teams. In the Virtual Lab, published in Nature by James Zou's group at Stanford, one language model acted as the principal investigator and led several others, each playing a scientist from a different field, through a series of research meetings. A human researcher gave only high-level feedback. The virtual team designed 92 nanobodies against recent SARS-CoV-2 variants. In lab tests, two of them bound more tightly to the JN.1 or KP.3 variant while still binding strongly to the original spike protein.

The result that drew the most attention came the same year from Google's AI co-scientist. José Penadés's group at Imperial College London had spent years trying to explain how a family of mobile genetic elements called cf-PICIs spreads between different bacterial species. The team gave the AI only published literature and the same question. In about two days it returned five ranked hypotheses, and the top one, that cf-PICIs hijack the tails of many different phages to widen their range of hosts, matched the mechanism the lab had confirmed experimentally but not yet published. The biology and the comparison with the AI were published together in Cell.

Generating hypotheses does not always mean a chatbot. AI-Descartes, described in 2023 in Nature Communications, combines logical reasoning with symbolic regression. Given a few axioms and a small amount of data, it rederived Kepler's third law, Einstein's time-dilation formula and Langmuir's adsorption theory. When the data alone could not separate candidate formulas, logic picked the one consistent with theory.

Gap one: sounding new is not the same as working

An idea is only as good as the result it produces. Chenglei Si, Diyi Yang and Tatsunori Hashimoto at Stanford tested this in two steps.

In 2024 they asked more than 100 natural language processing researchers to score research ideas without knowing which were written by experts and which by an AI. The AI's ideas scored higher on novelty and slightly lower on feasibility, a result that was widely reported as AI out-thinking humans.

The follow-up, posted in June 2025, actually carried the ideas out. Forty-three researchers were randomly assigned either an AI idea or a human one, and each spent more than 100 hours implementing it and writing a four-page paper. When the finished projects were scored, the AI ideas dropped significantly more than the human ones on novelty, excitement, effectiveness and overall quality, and the earlier advantage largely disappeared. The authors call this the ideation-execution gap.

Today's models are good at writing research plans that sound fresh. Reading the plan alone tells you little about whether it will pan out. One plausible explanation, which the study itself did not test, is that models have read a huge number of papers and are good at recombining ideas across fields, but have no feel for which combinations fail on real data. Those problems usually surface only once someone tries.

Gap two: checking its own results

In November 2023, a team from UC Berkeley and Lawrence Berkeley National Laboratory reported the A-Lab in Nature: an autonomous lab combining robots, machine learning and language models trained on the literature. Over 17 days of continuous operation, the paper said, it made 41 new inorganic compounds out of 58 computationally predicted targets.

A few months later, solid-state chemists led from Princeton went through the products one by one in PRX Energy. They found four recurring problems in how the data were analyzed, including automated refinement of powder X-ray diffraction patterns that was not yet reliable, and predictions that ignored disorder, where different elements share the same positions in a crystal. Their conclusion was that no new materials had in fact been discovered. In January 2026 the original authors published a correction in Nature that changed the title from "novel materials" to "inorganic materials" and revised how the novelty of the targets and the synthesis results were described.

Powder diffraction refinement is hard because a single pattern can often be explained, more or less, by several different structures, with impurity phases, disorder and measurement noise all mixed in. Choosing the right interpretation takes a chemist's judgment. The robots did run a great many experiments. What went wrong was deciding what had actually been made. Synthesis can be automated; reliably interpreting the outcome still takes experienced people.

Gap three: who asks the question

Look back at the successes and most of them start with a person. Penadés's group spent years narrowing the problem to the point where they could ask the AI a precise question. The Virtual Lab had a human steering it. Even in its most open-ended mode, the AI Scientist works only in machine learning, where an experiment is a piece of code and you know within minutes whether it ran.

Put together, these cases outline where research agents work best right now: the question is well defined, experiments can be run cheaply and automatically, and results are easy to verify. Machine learning research, cloud-lab chemistry and computational design followed by lab screening all fit. Animal studies that take months, early exploration where the hard part is deciding which question matters, and complex materials characterization where the data are hard to read still depend on people.

There is also the question of what should not be done. The ChemCrow team built safety tools into their agent because a system that can plan the synthesis of an insect repellent can also be asked to plan something dangerous. Deciding where those lines sit is another judgment that, for now, rests with people.

The AI Scientist's authors raise one more risk in their paper: systems that turn out large numbers of uneven papers could add noise to the scientific literature.

What this means for people who do research

For graduate students and working scientists, the change is likely to show up in the middle of the process first. Literature searches, experiment code, lists of candidate hypotheses and data cleanup are already being handed partly to agents. That could free up time, or it could simply raise expectations for how much each person produces; it is too early to say which. Deciding which idea deserves months of work, and figuring out what a result really means, is still a human job, and people still carry the cost when those judgments go wrong.


Further reading: The studies cited here appeared in Nature Machine Intelligence, Nature Communications, Nature and Cell, among others. For public data on journals in computer science and AI, see the CrushSCI computer science journal ranking.

References

  1. Lu C, et al. Towards end-to-end automation of AI research. Nature, 2026. doi:10.1038/s41586-026-10265-5
  2. Boiko D A, et al. Autonomous chemical research with large language models. Nature, 2023. doi:10.1038/s41586-023-06792-0
  3. M. Bran A, et al. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 2024. doi:10.1038/s42256-024-00832-8
  4. Swanson K, et al. The Virtual Lab of AI agents designs new SARS-CoV-2 nanobodies. Nature, 2025. doi:10.1038/s41586-025-09442-9
  5. Penadés J R, Gottweis J, et al. AI mirrors experimental science to uncover a novel mechanism of gene transfer crucial to bacterial evolution. Cell, 2025.
  6. Cornelio C, et al. Combining data and theory for derivable scientific discovery with AI-Descartes. Nature Communications, 2023. doi:10.1038/s41467-023-37236-y
  7. Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers. arXiv:2409.04109, 2024.
  8. Si C, Hashimoto T, Yang D. The ideation-execution gap: Execution outcomes of LLM-generated versus human research ideas. arXiv:2506.20803, 2025.
  9. Szymanski N J, et al. An autonomous laboratory for the accelerated synthesis of inorganic materials. Nature, 2023. doi:10.1038/s41586-023-06734-w (Author Correction, 2026, doi:10.1038/s41586-025-09992-y)
  10. Leeman J, et al. Challenges in high-throughput inorganic materials prediction and autonomous synthesis. PRX Energy, 2024. doi:10.1103/PRXEnergy.3.011002

By the CrushSCI editorial team. AI tools assisted with research and drafting; facts and citations were checked by a human editor.

1 Comment

0 votes
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

AAIF Hits 10 Projects. Its CTO on What to Adopt Now and the Security Gaps Left Open

Tom Smithverified - Oct 23

How AI Is Changing Web Development, and What Developers Still Need to Understand

Daniel Decker - Nov 10

CData's New AI Gateway Puts Context at the Center — Fixes the Three Reasons Enterprise AI Stalls

Tom Smithverified - Sep 28

AI Agents Have a Communication Protocol Now. They Still Don't Remember Anything.

Tom Smithverified - Sep 9

AI Agents Don't Have Identities. That's Everyone's Problem.

Tom Smithverified - Mar 13
chevron_left
133 Points • 3 Badges
1Posts
0Comments

Related Jobs

View all jobs →

Commenters (This Week)

1 comment
1 comment

Contribute meaningful comments to climb the leaderboard and earn badges!