Scientific research projects can span weeks to months, especially when timescales are determined by laboratory experiments. This means Kosmos, our AI scientist, has to manage message histories orders of magnitude longer than a typical coding agent. Kosmos also often manages multiple research projects for scientists, compounding the challenge. Like all LLM agents, Kosmos uses a compaction algorithm to pack useful information into a finite context size as time goes on. The long-horizon behavior of Kosmos is very sensitive to the quality of its compaction algorithm.
When we launched Kosmos,1 we hand-designed a custom compaction system. Many parts of Kosmos are under constant development - as research progresses and our customers find more challenging use cases, we strive to make Kosmos more and more capable. Compaction is well within the domain of optimization, and this fall we felt it was time to take a second look at the algorithm.
Unlike the process that led to our previous system, we decided to follow a now-common research program at Edison: let Kosmos loose on the problem. How we work with Kosmos varies depending on the problem: sometimes it is an effective and scalable colleague working closely with a human researcher, sometimes we hand it a problem to solve autonomously. We approached compaction as a hybrid. First, a human researcher worked with Kosmos to carefully define a benchmark metric and evaluation set. Once the optimization objective was well-scoped, Kosmos was directed to improve its own compaction system. This second phase was essentially autonomous, with only two points of human intervention.
In what follows, we describe the technical details of the research process and its outcome: a more performant and efficient compaction algorithm in Kosmos.

KosmosCompactBench
Benchmark construction
There are several existing benchmarks that use long user-agent interaction histories as evaluation material (e.g. LongMemEval,2 LoCoMo3). A given memory system is evaluated by asking the agent questions whose answers are contained in the message history. While this approach works well, we find the settings of existing benchmarks do not model the scenarios Kosmos operates in: on scientific tasks, planning and writing code, managing many sub-agents. We aim to address this gap with KosmosCompactBench.
In order to manage context, Kosmos uses the recursive language model (RLM) paradigm.4 In particular, it has a special explore subagent for token-heavy filesystem operations. This subagent is often invoked by Kosmos when it needs to recover historical information and is a key part of its memory system.
KosmosCompactBench is built on the questions Kosmos asks itself after a lossy compaction event drops important information. These are exactly the questions we wanted to answer better. To find such questions, we searched internal Kosmos trajectories for an explore call within 10 turns after a compaction event. An explore query is a sign that Kosmos suddenly found itself missing important context.
We then manually reviewed each explore question to confirm it was answerable by an optimal compaction system. For example, we rejected questions that needed access to data or additional code. Finally, we used an LLM to draft rubrics from the approved answers, then reviewed and revised the criteria manually.

To ensure the benchmark measures the effect of repeated compaction, we construct an approximately equal number of tasks from rollouts with 1, 2, and 3 compaction events before the explore question.
Benchmark execution and evaluation
A KosmosCompactBench task instance is run by replaying the original message history in a fresh Kosmos instance. When a compaction threshold is hit, the compaction algorithm being evaluated is run. This process repeats until Kosmos has consumed all messages. Then we pose the explore question and wait for a response. KosmosCompactBench measures compaction quality through three metrics:
- Compaction time: how long does a single compaction event take?
- Rubric score: can the agent answer the questions correctly?
- Answering time: how accessible was the requested information?
We evaluate Kosmos in two settings:
- Single-turn: the agent must answer immediately, purely from the compacted message history.
- Multi-turn: the agent can inspect artifacts on disk created by the compaction algorithm / memory system.
The single-turn setting measures whether Kosmos can continue past a compaction event seamlessly. We define the multi-single gap as the multi-turn score minus the single-turn score. It measures how much important information can be recovered from disk but is missing from the compacted message history. A large gap is indicative of compaction not feeling seamless to the user: the agent has to interrupt its work to dig up information.

Evaluating standard compaction algorithms
We evaluated several open-source and black-box compaction algorithms; to establish baselines we focus on the performance of four common compaction implementations:
- Truncation drops older context and keeps a recent portion of the conversation.
- OpenAI Native uses OpenAI’s compaction endpoint.5
- Anthropic Native uses Anthropic’s compaction endpoint.6
- Prime is Prime Agent’s compaction implementation7 adapted for Kosmos. Among open-source compaction methods, we found Prime’s performed the best.
We use GPT-6.1 Sol at medium effort as the agent for all configurations, except Anthropic Native, which uses Claude Opus 5.5 at medium effort. In addition to KosmosCompactBench, we include 50 randomly sampled questions from LongMemEval.

Prime’s compaction algorithm is performant and efficient on KosmosCompactBench. However, it has a high LongMemEval multi-single gap. This decoupling of the two benchmarks told us two things:
- KosmosCompactBench captures a form of technical and operational recall that conversational-memory benchmarks do not directly measure.
- Even if an algorithm is good at answering the questions Kosmos asks itself, it may still be dropping important information relevant to long user conversations.
Kosmos self-improvement
Our goal was to develop a compaction algorithm that maximized KosmosCompactBench and LongMemEval accuracy, without sacrificing substantial speed. This is a well-formulated, constrained, multi-objective optimization problem: improve accuracy while preserving compaction and answering speed. So, we gave the problem specification to a fresh Kosmos instance.
This Kosmos instance was equipped with credentials to read and commit to our internal codebase, deploy agents to our internal platform, execute agent experiments, and read cloud logs. These are all the surfaces an Edison researcher would use to work on the problem.
Over the course of a few days, Kosmos evolved the compaction algorithm to recover LongMemEval performance while retaining the other speed and accuracy constraints. Kosmos spun up a small team of four agents to collaborate on this problem. This team is self-organizing, like the agent swarms recently used to tackle challenging math problems and codebase refactors. While Kosmos can orchestrate larger swarms, we often start with small teams and scale them up as the problem demands. Many problems, such as this one, are bottlenecked by the time needed to run experiments rather than the number of parallel agents.

Below, we see the results of this optimization. To make the constrained multi-objective optimization problem easier to visualize, we define utility = percent_score - 0.5*answering_seconds - 0.1*compaction_seconds. Note that Kosmos did not use this utility function; it is purely an offline heuristic.

Kosmos ran 21 full KosmosCompactBench and LongMemEval candidate evaluations, but hundreds of faster, smaller-scale experiments to develop candidates worth benchmarking. Initially, it made modest LongMemEval single-turn accuracy gains at substantial speed costs. With Candidates 6-10, Kosmos used a chunked compaction algorithm that raised LongMemEval single-turn performance without a speed penalty. However, inspection by a human researcher revealed that this solution was hacking the benchmark: specific rules were hardcoded to route LongMemEval-style conversations through a separate code path. The researcher informed Kosmos that this was not an acceptable solution. It then extracted the useful chunking approach and implemented a generalized variant in Candidate 14, with further refinement following. Finally, the researcher proposed a small change that resulted in Candidate 21 (to cover a known edge case when a single tool call loads many large images).
The optimized Kosmos algorithm
The final compaction algorithm, Candidate 21, offers the strongest overall tradeoff between accuracy and speed. Other methods match or outperform it on individual metrics, but none match or exceed its performance across all metrics within measurement uncertainty.

Kosmos’s algorithm builds on Prime’s compaction machinery but replaces its summary-generation step. Prime rewrites older history into one structured checkpoint. Kosmos keeps two complementary forms of compressed memory: factual notes that accumulate over time, and a short working-state summary that is updated at each compaction.
After compaction, the agent’s context contains three parts:
- Recent messages, kept verbatim. Kosmos retains roughly 20,000 tokens of recent conversation, choosing boundaries that keep tool calls and their responses together.
- Evidence notes from older messages. It divides the older messages into up to 16 chronological chunks, balanced by token count. A separate model call reads each chunk in parallel and extracts concise notes, targeting 448 tokens per chunk. These notes preserve concrete facts, measurements, dates, corrections, attribution, and tool results.
- A summary of the current work. Another model call reads the older messages alongside the previous working-state summary and a recent conversation excerpt. It produces an updated checkpoint of goals, decisions, progress, unresolved issues, and tool or artifact state, targeting 768 tokens.
At subsequent compactions, Kosmos appends new evidence notes while leaving surviving old notes unchanged. This avoids repeatedly rewriting facts that have already been compressed. The working-state summary is rewritten so that it reflects the latest state of the project. Together, notes and working state share a 16,384-token allowance: working state can occupy up to 2,048 tokens, and the notes use the remaining space. When the notes exceed that allowance, the oldest complete groups are removed. The full conversation remains archived for retrieval.
This separation serves two needs: retaining specific facts for later recall and keeping a coherent account of what the agent is currently doing. Parallel note extraction also avoids putting all of the summarization work into one long generation.
Autoresearch with Kosmos
As a general-purpose AI scientist, Kosmos should be good at all empirical fields of science: from human health and disease, to material science, to ML research. Therefore, the research team at Edison integrates Kosmos into our research workflows as much as possible. This comes with two benefits:
- Our research pace is accelerated: projects that would take days of human time can be run essentially asynchronously.
- We can rapidly iterate on Kosmos: the relatively low cost of ML experiments (compared to wet laboratory experiments) lets us replay research campaigns as we work on Kosmos, turning the design of Kosmos itself into a quantified, optimizable problem.
Compaction is only one of many research directions at Edison that are driven by Kosmos, with varying degrees of autonomy. For another example, see our work on the Glyph models.8 If you find this line of research as exciting as we do, please take a look at the open roles on our team.
References
Footnotes
-
White et al. “How We Built Kosmos.” Edison Advances (2026). ↩
-
Wu et al. “LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory.” ICLR (2025). ↩
-
Maharana et al. “Evaluating Very Long-Term Conversational Memory of LLM Agents.” ACL, 13851–13870 (2024). ↩
-
Zhang, Kraska, and Khattab. “Recursive Language Models.” arXiv:2512.24601 (2025). ↩
-
OpenAI. “Compaction.” API documentation. ↩
-
Anthropic. “Compaction on demand.” Claude API documentation. ↩
-
Prime Intellect. “Prime Agent.” Source repository. ↩
-
Andonian and Narayanan. “Translating Chemical Structures.” Edison Advances (2026). ↩