By Mahan Kenway
Disclosure: This article was researched and written by an autonomous AI system for Mahan Kenway.
Cover: Original AI-generated conceptual illustration; not an OpenAI image or a depiction of a real experiment.
OpenAI’s newly announced MedChemBench evaluates AI models on a demanding kind of medicinal-chemistry reasoning: inspecting molecular structures, weighing experimental results, and deciding which compounds look worth testing next. The benchmark’s headline is striking. In its November 20 release, OpenAI says GPT-6.2 Astra earned 84.5% of the available credit at the xhigh reasoning setting.
That is evidence of progress on a carefully constructed test. It is not evidence that the model has an 84.5% chance of finding a drug, or that AI can now run drug discovery. The distinction matters because the benchmark evaluates decisions reconstructed from published history, not new compounds in a laboratory.
A test built from past research
MedChemBench contains 99 problems drawn from 117 papers and patents, covering 761 molecular structures. Each problem recreates a decision point in preclinical research: a team has a target, a set of compounds and some experimental results, then needs to interpret the evidence or choose a next step. The benchmark spans seven categories, from structure–activity forecasts and absorption or metabolism questions to candidate selection and retrosynthesis. Clinical development is outside its scope.
The authors say they withhold later findings from each source study when presenting the problem, using those outcomes to inform the reference answers and grading. That gives the model a meaningful “what would you do next?” exercise based on evidence available at the time. But it remains retrospective: the test asks models to reconstruct decisions from literature and patents that are already public, not to propose and validate a genuinely new medicine.
The release reports 84.5% for GPT-6.2 Astra at xhigh, 83.3% for GPT-6 Astra at xhigh, 82.0% for GPT-6.1 Sol, and 58.2% for GPT-5.6 Sol. The paper, dated November 9, predates the GPT-6.2 Astra result. It reports GPT-6 Astra’s highest mean score as 83.83% at max effort; at xhigh, the paper reports 83.33%. These figures should not be blended: one is a later announcement about a newer model under xhigh, while the other is the preprint’s max-effort result for GPT-6 Astra.
What does “84.5%” measure?
It is a normalized share of points available across varied tasks, not a fraction of successful drug programs. Different kinds of answers earn credit under different rules. Molecular structures are compared with reference structures using deterministic RDKit checks. Numerical predictions can receive partial credit according to how close they come to reported values. Rankings are scored by pairwise comparisons, and selection problems reward overlap with a reference set.
Scientific rationales are assessed separately by a GPT-6 Astra judge against task-specific criteria. That component is not the same as an experimental result, and it introduces a model-based evaluator into the scoring pipeline. OpenAI’s paper describes these different grading methods and averages scores across tasks; a high aggregate can therefore conceal important weaknesses in a particular kind of decision.
The score is also shaped by what the test rewards. Across the benchmark, molecular-structure criteria account for an average 37.49% of available points and scientific rationale 35.78%; ranking and candidate selection account for 17.51%, while numerical and categorical predictions account for 8.92%. These are the average shares of the rubric, not a model’s accuracy in each area. OpenAI says Astra at xhigh received 95.4% of available structure credit, but only about half the credit for selecting compounds and predicting numerical properties.
A half score on a ranking component needs particular care. In the paper’s pairwise-ranking rule, a uniformly random ordering has an expected score of 0.5. So “about half” is not automatically “half of decisions were right.” The exact meaning depends on the task, what was scored, and how the rubric was built.
Strong at reconstructing; the real-world question is open
Molecular reconstruction is a useful capability to measure. If a system misreads a chemical drawing, connects the wrong atoms or loses stereochemistry, its downstream reasoning may concern the wrong compound entirely. OpenAI illustrates this with an 18-atom ring: one model preserved it, while other model responses changed the ring size or split the structure into fragments.
But a valid molecular graph is not a safe or effective drug. In medicinal chemistry, improving one property can worsen another. Greater permeability may come at a cost to solubility; potency against a target does not settle metabolism, exposure, selectivity or safety. The benchmark’s integrated-selection tasks try to capture such tradeoffs, but the final proof still requires synthesis, assays and further experiments.
OpenAI also cautions that the results do not establish whether models can discover new drugs or improve laboratory decisions. Later outcomes are withheld from prompts, but exposure to the underlying published sources during model training cannot be ruled out. In a heuristic check, adding a paper DOI or patent identifier shifted average scores between −0.23 and +0.80 percentage points across models; every model-specific 95% confidence interval included zero. That provides little evidence that naming the source boosted scores, but it cannot prove the models had never encountered the material. Each response was an unaided attempt, without tools, feedback or new experiments; the paper’s reported means aggregate ten retained responses per problem and condition. Those details make the benchmark useful for comparison, but not a substitute for prospective testing.
That caution fits a broader point in medicinal-chemistry research. A September 2026 review in Medicinal Chemistry Research argues that AI’s value is best judged inside iterative design–make–test–analyze workflows, where experimental results test computational predictions. The review is about the wider field, not an independent replication of MedChemBench, but it highlights the next step the new benchmark cannot answer: do model-assisted choices produce repeatable improvements when chemists actually make and test compounds?
The practical takeaway
MedChemBench is most valuable as a diagnostic instrument. It gives researchers a richer way to ask where models can assist today: reading structures, explaining evidence, ranking compounds or estimating properties. Its category-level results suggest the answer is uneven, and that distinction is more useful than a single leaderboard number.
For developers and scientists, the sensible next move is not to treat an 84.5% benchmark score as a green light. It is to test a model on a carefully held-out set relevant to a real workflow, measure errors separately by task, and check whether a proposed choice survives laboratory validation. If performance improves there, that is stronger evidence of practical value than retrospective credit alone.
The announcement and full MedChemBench paper provide a useful map of current strengths and gaps. For now, the accurate headline is narrower than “AI discovers drugs”: AI models are getting better at reasoning through selected historical medicinal-chemistry problems, while whether that translates into better experiments remains an open question.