Writing First: How We Designed the Research-Agent Harness That Won Ralphthon@ICML

Team WooandB: Woomin Song and Byungjun Yoon

Ralphthon@ICML is a hackathon where AI agents conduct end-to-end research under strict time constraints. Teams receive 1.5 hours to configure their Research Agents, followed by a three-hour autonomous run covering topic selection, implementation, experimentation, analysis, and paper writing.

Human intervention is allowed, but anyone who directly instructs the agents must wear a ridiculous lobster costume provided by the organizers.

Our team, WooandB, consisted of Woomin Song and Byungjun Yoon, Ph.D. students in Professor Jinwoo Shin’s lab at KAIST. We were fortunate to win first place in the AI Scientist track.

The most important decision we made was not how to implement the method or which experiments to run. Instead, we spent much of the setup period answering a more fundamental question:

What should a good research paper look like?

This question became the foundation of our Auto Research Agent Harness.

We required the agents to maintain a complete, submission-shaped manuscript from the beginning, rather than asking them to experiment first and write only if they found something good. The paper defined which experiments were needed, captured the current scientific hypothesis, and forced the research direction to change when the evidence no longer supported the existing story.

This post describes that design and the lessons we learned. A follow-up post will examine the agents’ actual research trajectory in detail.

Choosing the research topic with a single question

Since we had no idea what to work on, our team discussed it with ChatGPT.

At 10 AM on the day of Ralphthon, we asked Woomin’s personal ChatGPT account:

We have to write an ML paper in three hours, from ideation to experiments. What would be a good topic? It needs clear novelty, but the experiments must also be fast enough for multiple iterations.

We gave ChatGPT two requirements:

1. The idea should have clear novelty rather than being a minor variation of existing work.
2. The experiments should be fast enough to support several iterations within three hours.

Nowadays, most top AI services manage personalized memories of your conversation history. From there, ChatGPT drew on research ideas and problems we had previously discussed — though none of them specifically for Ralphthon. It combined several of those contexts and proposed Depth-AR. We selected its first suggestion.

A short description of our topic, Depth-AR: we consider Transformer depth as an optimization trajectory and predict a skipped layer’s residual update from the updates in preceding layers. The goal was to test whether information lost through layer skipping could be partially recovered without additional training.

The important part was not providing a long technical prompt. It was clearly defining what made a topic suitable for this particular setting.

We applied the same principle to the agent system: specify the desired outcome first, then let the agents determine the steps required to reach it.

The core design: write the paper first

The central principle of our harness was Writing First.

A conventional workflow might look like this:

Idea → Implementation → Experiments → Analysis → Paper

We reversed it:

Idea → Submission-shaped draft → Required experiments → Experiments → Paper revision

Once the topic was selected, the agents first defined:

• the paper’s central claim,
• the experiments required to support it,
• the question answered by each table and figure,
• the expected storyline,
• and alternative interpretations if the results differed from expectations.

Experiments that had not yet been run were represented with placeholders. These were clearly marked in the source so that the agents would not confuse them with real results, and every placeholder had to be replaced before submission.

Out of context, “write the paper before the results exist” sounds like fabrication. It is the opposite: a write-ahead log with mandatory verification. During the run the agents split it into two LaTeX macros with deliberately opposite rules — \ph{} for an invented number, which must either be replaced by a real measurement or take its sentence down with it, and \phm{} for a number a result file already backs, which must never be silently dropped. ralph/PH-LEDGER.md maps every provisional value to the exact JSON key it is waiting on, and verify-phm.py and writing-audit.sh fail the build if anything invented survives to submission. Nothing made up can ship, and nothing measured can quietly disappear.

This gave each experiment a concrete purpose. Rather than vaguely checking whether the method worked, the agents knew which claim or missing comparison the experiment needed to address.

It also made the system responsive to unexpected results. When evidence contradicted the current hypothesis, the agents revised the claim and storyline before deciding what to run next.

During the three-hour run, the paper’s central storyline changed at least six times. Each shift is a logged decision with the evidence that forced it, in ralph/DECISIONS.md — the final restructure is labelled, literally, storyline v6.

The agents did not finish the experiments and then search for a story. They continuously revised the story, and the current story determined the next experiment.

Defining what a good paper looks like

We spent more than 30 minutes of the first 1.5-hour human setup period building a clear, general definition of what counts as GOOD research and a WELL-WRITTEN paper.

These guidelines were not specific to Depth-AR. They described principles that could apply to almost any research project:

• Focus on one concrete research question.
• Connect every experiment directly to a claim.
• Give every table and figure a clear takeaway.
• Explain what the results mean rather than merely listing them.
• Revise the storyline when results differ from expectations.
• Keep the draft in a hackathon-submittable state throughout the run.
• Place the strongest results in the main paper rather than hiding them in the appendix.

Both documents are in the repository verbatim, and both are project-agnostic — nothing in them is specific to Depth-AR. writing-guidelines.md is the process: what to write, in what order, and what “submittable” means as a checklist. Its first rule is the whole harness in one line — “The paper must be submittable at every stage. Not ‘finished’ — submittable.” writing-style-guide.md is the form: the page budget section by section, the two-part shape every caption follows, \citep over bare \cite, and the rule that no claim ships without a number next to it.

We then gave these principles to the same ChatGPT session that had proposed Depth-AR. ChatGPT adapted those general guidelines for research and writing into a topic-specific research and writing plan, which became the initial instruction document for the autonomous agents: DEPTH-AR-PLAN.md. The part of it that mattered most turned out to be the pre-committed pivots — a paper storyline written in advance for every outcome, including the one where the central hypothesis fails outright. When it did fail — fourteen minutes into the run, with the measured effect coming out anti-momentum, the exact opposite of the premise — the agents did not have to improvise what a disappointing number meant. The plan had already decided.

The humans defined the standard for a good result. The model converted that standard into an executable plan for the specialized topic.

Dividing the work among four agents

We used four agents so that orchestration, experiment, and writing could proceed in parallel.

The role names above link to the actual persona files — they are the harness, and they are short enough to read in a sitting. (Three of them, strictly: the monitoring role ran as a watcher process and a ten-minute rethink loop rather than as a fourth persona file.)

One rule underneath all three did more work than any prompt wording: anything crossing an agent boundary is a file on disk, never a chat message. The Writing Agent reads numbers directly out of the result JSONs. A number relayed through conversation is treated as corrupted by definition.

The goal was not simply to use more agents.

A single agent can become absorbed in debugging while the manuscript remains untouched. Conversely, an agent focused on writing may leave the GPUs idle. Separating the roles allowed experiments and manuscript updates to continue in parallel while reducing the context each agent had to manage.

The role instructions themselves were approximately as simple as the table above.

We also gave the Main Agent an Anthropic blog post about multi-agent systems and asked it to apply any relevant ideas. We did not have enough preparation time to study and operationalize the post ourselves.

What we provided

Our initial setup consisted of:

• general guidelines for a good research paper — writing-guidelines.md and writing-style-guide.md,
• the four-agent structure — master.md, experiment.md, writing.md,
• a project-specific research and writing plan — DEPTH-AR-PLAN.md,
• skills for configuring the GPU environment,
• a Git-connected Overleaf project,
• and a herdr-based communication skill for sharing state and results across agents.

Most of the detailed plans, code, experiment scripts, tables, and manuscript text were created by the agents. Every file in the harness repository except its README was written by the agents themselves.

We read the paper, not the logs

A little over an hour after the autonomous run began, we returned from lunch and opened the latest paper.

The agents had already produced a coherent basic paper. The research question and method were clearly described, and actual experimental results had been inserted into the tables.

It was a meaningful research artifact under the hackathon constraints, though not at the level required for submission to a conference such as ICML.

Because the manuscript had been maintained throughout the run, we did not need to reconstruct the project from long agent logs. The paper already represented the current state of the auto research.

We read it and provided high-level feedback:

• Use the available GPUs more aggressively.
• Organize the paper around where meaningful gains appear.
• Move the strongest result from the appendix into the main paper.
• Sharpen the central message connecting the results.

This resembled an advisor reviewing a student’s draft. We did not supervise every command or prescribe every experiment. We commented on the scientific direction and priorities.

A few rounds of this feedback were enough for the agents to reprioritize their experiments and substantially improve the paper.

Takeaway 1: define the desired outcome first

What we expect from an auto research agent is a conclusive scientific finding with a conclusion — not an ablation study.

We therefore focused on defining the final output rather than specifying every intermediate action. The same principle guided topic selection: novelty mattered, but so did experimental speed and the ability to iterate within three hours.

Ambiguity in the objective propagates through a long-running workflow. When the desired final state is concrete, agents are better able to determine which intermediate steps matter.

A good harness begins with a clear description of the goal — what success looks like.

Takeaway 2: long-running agents need an anchor

Research Agents can easily become absorbed in implementation details, minor observations, or low-priority experiments. They may also lose track of earlier decisions after context compaction.

In our system, the manuscript — the draft paper — served as an anchor.

Every new result had to be connected to the paper:

• Does it support the central claim?
• What part of the manuscript should change?
• Does the storyline still make sense?
• Which experiment is now most important?

Maintaining a submission-shaped draft repeatedly brought the agents back to the big picture.

Humans use drafts and research notes in much the same way. Long-running agents also benefit from an external artifact that connects local actions to the final goal.

Takeaway 3: a harness should improve through use

Several interventions we made during the run could have been included in the initial harness:

• Check whether compute resources are idle.
• Adjust experiment scale based on the remaining time.
• Build the paper around the strongest supported pattern.
• Ensure that the strongest result appears in the main text.
• Verify consistency across the text, tables, and appendix.

We recognized these needs only after observing the system.

Some of this happened during the run itself. The persona files started the day as plain role descriptions and ended it with eleven numbered rulings appended to them — banned softener phrases, dtype-aware tolerances, a visual gate requiring that a figure be looked at before it ships, and cite by identity, never by value — each written the moment an incident exposed the gap. Reading the rules sections of master.md against the incidents in ralph/DECISIONS.md that produced them is the clearest record we have of a harness learning from itself.

In the next version, the rest can become initial instructions or automated checks. New runs will expose new failure modes, which can then be incorporated into the harness.

A harness should not be treated as a prompt written once and left unchanged. It should evolve by converting recurring human feedback into reusable rules, checks, and skills.

Takeaway 4: autonomous research works, but for now the AI co-scientist is better

A single Ralphthon run does not show that agents can independently produce mature scientific discoveries. Ralphthon had strict limits on time, compute, and deliverable format.

However, it did show that an end-to-end research loop can work. The agents selected a topic, implemented a method, ran experiments, interpreted the results, revised their hypotheses, and wrote a paper.

Still, the strongest result did not come from autonomous execution alone. Human judgment remained valuable for deciding which findings mattered, which unexplored directions were worth pursuing, which experiments deserved the remaining time, and how the evidence should be framed.

At the same time, humans did not need to supervise every line of code or inspect every log. Reading the manuscript and adjusting the high-level direction was enough to produce a substantial improvement.

For now, we believe Research Agents are most powerful as co-scientists: agents rapidly implement, experiment, analyze, and revise, while humans periodically provide scientific judgment.

Closing remarks

Ralphthon was an exciting opportunity to test how much of the research process an agent system could complete within a few hours, and we had a lot of fun!

We thank Team Attention and the event organizers, as well as the sponsors, including OpenAI, Weights & Biases, VESSL AI, DALPHA, and NAVER D2SF.

We have open-sourced everything: the full harness, and the paper together with its complete research trace — the decision ledger, every result file, and the agents’ own commit history, with credentials removed.

In the next post, we will examine the research process itself: the hypotheses the agents formed, how they interpreted failed experiments, and how the paper’s storyline changed during the three-hour run.

We plan to continue developing and releasing autonomous Research Agents and human-AI co-scientist systems.