Improving Scientific Writing by Learning from Good Examples

Motivation

Reinforcement learning-based post-training of LLMs typically relies on the ability to discriminate better versus worse LLM outputs. RL with human feedback (RLHF) and direct preference optimization (DPO) require humans to supply preference rankings, RL with verifiable rewards (RLVR) uses a “verifier” - a (often binary) reward function that can judge the correctness of an output.

At Edison Scientific, we work on training systems and LLMs to conduct challenging scientific research tasks. Some of these tasks are verifable, such as those in LabBench2¹ or BixBench3². However, many critical tasks do not easily admit rubrics or determinstic reward functions. Consider writing scientific documents - while an expert in a field can determine if a paper, patent, or regulatory document is well-written, they would be hard-pressed to enumerate all of their evaluation criteria.

Such a task may seem ideal for RLHF, but RLHF requires a large number of human evaluators to provide training signal. This can be especially challenging when judging scientific writing, when careful evaluation of the content and writing style requires more than a cursory read. On the other hand, we have millions of examples of good scientific writing in the public domain: papers, patents, etc. We were motivated to develop a method that would teach LLMs to learn directly from positive demonstrations, without requiring the construction of synthetic negatives, auxiliary reward functions, or additional labels.

Method

We started with an idea that has a long lineage: adversarially training a policy and a reward function. Learning a reward from demonstrations is the premise of inverse reinforcement learning,³ and adversarial methods such as GAIL⁴ and AIRL⁵ made it work for robotics and control. Early attempts to adapt it to language⁶ came before modern reasoning models (and would require examples of good reasoning traces to apply to reasoning models). We bring that line of work to modern reasoning models and demonstrate its effectiveness at scientific writing. We refer to our method as Reinforced Adversarial Learning (RAL).

We are not alone in revisiting adversarial training for LLM post-training. We detail the differences between RAL and prior/concurrent methods in Section 2 of our paper.

RAL trains two models together. The generator produces a reasoning trace followed by a final answer. The judge sees only the final answer, never the reasoning, and estimates the probability that it came from the reference set, a collection of real examples of good work, rather than from the generator. That probability is the generator’s reward. The generator is trained with GRPO⁷ to maximize the probability of fooling the judge, while the judge is trained with a cross-entropy loss to tell references from generated answers, so the two improve as adversaries. Because the judge scores only the final answer, the reasoning itself is never rewarded. A reasoning trace pays off only when it leads to an answer the judge finds convincing, so the generator must develop reasoning that actually works rather than reasoning that looks right.

The two models improve together, each moving the target for the other. Co-training two models this way can be unstable. To keep the judge from becoming too strong too quickly, we skip its updates when its accuracy exceeds a threshold, preserving a useful learning signal for the generator. We also keep a replay buffer of past generations and mix them into the judge’s training batches, so it does not forget strategies the generator has already tried.

Integer multiplication 4137 × 8264 = ?

Before turning to scientific writing, we check that the judge can learn a meaningful reward on a task where we know the right answer. A natural worry is that the adversarial game stays superficial, with the judge telling the two distributions apart by length, formatting, or word choice. We therefore start with a reasoning-heavy task whose answers offer no such cues, multiplying two four-digit integers. To get the product right, the generator must break the problem into steps in its reasoning, such as partial products, before committing to an answer. A product is either right or wrong, so the judge can separate references from generations only by learning what makes a product correct. A verifier withheld during training measures the result.

With enough judge capacity and co-training, RAL matches or outperforms training against the verifier directly, and larger judges learn rewards that track correctness more closely. Today’s LLM judges can therefore look past surface differences. Our paper reports the same pattern on two more verifiable tasks.

Writing when there is no verifier

We then return to the scientific artifacts that motivated this post, whose quality experts can assess but no simple rule can capture. For this demonstration, we limit ourselves to scientific abstracts. In practice, an abstract reports results from the paper or proposal behind it. For simplicity, we do not tie the generator to specific results. Conditioning the generator on real results, and showing them to the judge, is a natural next step.

Our first task asks the generator for a credible research-paper abstract and specifies no field. Every reference abstract, however, comes from a biomedical paper. The judge must notice this from the examples alone and steer the generator toward biomedicine while keeping its outputs diverse. This tests whether a learned reward captures what the references are about, not only how they are written.

A frozen judge (Claude Haiku 4.5) fails at this, even with more help than ours. We showed it each generation beside a real abstract, told it the target domain, and asked it to pick the real one. The generator earned a reward of 1 whenever Haiku picked its abstract, and 0 otherwise. The generator’s reward climbs steadily, yet its abstracts never become biomedical. Instead, it exploits the judge. Nearly every generation converges on a single narrow topic that the frozen judge keeps mistaking for real, even though it lies far outside the field. Because the judge never updates, nothing stops the exploit. RAL’s judge, Qwen3-14B, keeps learning, so an exploit stops paying off once the judge catches it. Given no hint of the domain, the co-trained judge still steers almost every generation into biomedicine. Diversity dips early, then recovers to its starting level.

NIH grant abstracts

Finally, we turn our method to writing abstracts for U.S. National Institutes of Health (NIH) grant proposals. Given only a proposal title, the generator must write a plausible abstract. The judge learns from real abstracts of grants funded in fiscal year 2026, chosen to reduce the likelihood of overlap with the underlying models’ training data. It must learn both what makes a credible NIH abstract and whether the abstract fits the proposal’s title.

Both an ensemble of frontier LLMs and biomedical experts mistake RAL abstracts for real ones more often than they do abstracts from frontier-model baselines. RAL reaches a mean fooling rate of roughly 35% against the LLM judges (top middle), compared with at most 4% for the baselines, and 55% against human experts (top left), compared with 21–37%. The top-right plot suggests how this progress unfolds. Judge accuracy repeatedly drops and recovers on fresh generations (red) while remaining high on replayed generations (blue), consistent with the generator finding new strategies and the judge learning to recognize them without forgetting earlier ones.

The excerpts at the bottom show what changes. Abstracts from the start of training are generic. By the end, they build a scientific argument, with a hypothesis motivated by preliminary findings, aims designed to test it, and experiments that rule out alternative explanations. The reasoning traces change too. They grow more specific and shape the final abstract more closely, even though the judge never sees them. We hypothesize that LLM judges also detect subtler differences that humans miss, which could explain their lower fooling rates, though we have not tested this explanation directly. This task produced the abstracts you can compare at the top of the page.

AI Research at Edison

At Edison we build Kosmos,⁸ an AI Scientist that reads literature, analyzes data, generates hypotheses, and executes scientific workflows. Our ML research starts from the problems we meet while making Kosmos a better scientist, and RAL grew out of one of them. We are hiring people to work on these problems. If this line of research interests you, take a look at the open roles on our team.

References