early results on autoresearch steerability

#What have we found so far?

1. Attractor basin/attractor spaces whenever we run autoresearch
   1. This attractor-basin is driven by the task; and is hard to move via levers which we have tried.
2. Different levers we have tried
   1. Same Prompt vs Repharsed Prompt
      1. Few semantic/word changes to the prompt, analyzed via DsPy
   2. Web search vs No Web Search
   3. Including domains (like neuroscience, geology, political science) don’t work.
   4. RTI - Reverse Translation Index/Indicator….
      1. We have three RTIs: Source, Implementation and Proposal.
         1. Source RTI: Checks whether a domain/paper-inspired proposal is actually faithful to its claimed source.
         2. Implementation RTI: Checks the code after Claude implements the frozen proposal.
         3. Proposal RTI: Checks the proposal before implementation. (is it specific or very template driven)

All of these results sort of point towards the same thing; which is that autoresearch is very task-driven , and based on the task, the search-space is sort of made.

#Same Prompt vs Rephrased Prompts Results

There are some interesting observations here:

1. The pretty obvious one is that as the steps increase, the performance also increases (with House-Price, Run-1 being the sole exception).

   1. Another interesting thing is in the middle of the runs (usually 8-14 steps), the agent tends to sort of “relax” or rather scores become more stable, meaning the agent is trying safer moves to move the benchmark score minimally[1].

2. Since none of the runs have memory enabled and new folders are made for each run (ensuring no leakage of context is happening), we can’t realistically say that run-1>run-2 (or vice versa). The next question which naturally arises is how predictable are idea probes to the actual runs.

   

3. Quite predictable, we can see that the autoresearch loop mostly implements mechanisms the 100-idea probe already surfaced. I believe the top-20 prior is a much better “view” of looking at things, since we only did 20 runs.

   1. One caveat which could be said is that we are using mechanism buckets, so basically “Of the interventions the loop actually accepted, what fraction had their mechanism bucket already listed in the probe's top-20 (or top-100)?" is the question which the above figure is answering.
   2. Major visual notice is how a lot of interventions were accepted in babylm and arc-agi-1, but not a lot on house-price. The explanation for this is because of the broader labels for house-price vs the other two tasks[2].

4. More interesting observations with the labels overlap.

   

5. We can see that “wordings” tend to shuffle across the idea probes (initial “give 100 ideas” prompt), but they tend to fall under same/similar mechanism buckets.

   1. For example; in babylm there are labels like: linear_warmup_cosine_decay, full_cosine_decay_to_zero_min_lr, sqrt_batch_lr_scaling, but technically all of them fall under the same mechanism bucket of lr scheduler tuning
   2. There are similar results in house-price and arc-agi-1 as well, but house-price tends to have much broader labels.
   3. Codex’s opinion on the above here

#Web Search vs No Web Search Results.

BabyLM results not here yet, since the metrics were different. But we can essentially see that web-search DOES NOT lead to better result.[3]

The main takeaway here is that web-search could have been a really good hillclimber-MAXXING tool for autoresearch, as it lets the agent just search for the best solutions or previously won solutions by different individuals; leading to a better starting point but its just too noisy and it spends more time searching and failing to implement them properly.

This was quite surprising to me because one would assume web-search would lead to better scores (as it can just access good solutions), but nah.

Another interesting observation here is that the web-searches being done by the agent IS DRIVEN by the prior. So it’s safe to say that web search mostly fetches sources to justify a mechanism the agent was already going to try, not new mechanisms outside the prior.

#Domain Runs

I haven’t ran the same for BabyLM and ARC-AGI (without RTI, yet), but the results are pretty much the same.

For a ML-driven benchmark; like house-price, vanilla ML will lead to better interventions compared to domains.

We also notice two more things here:

1. How much autoresearch sucks (or struggles) to translate domain knowledge properly
   1. This struggle was the reason for my RTI implementation.
2. And how this struggle of translation also means autoresearch just implements domain with a ML-taste.
   1. Hence the reason for making the non-ml envs benchmark . (Shourya ’s Idea)

With this result, we have proven that 3 levers which do not help in steering autoresearch to better interventions or ideas. But another question arises which is: to maximize efforts or scores on a benchmark, do we need novel ideas AT THE BEGINNING?

I’ve been thinking about this question for quite a bit of time, because the first few iterations or turns on a benchmark are usually very common and tend to boost the scores right away. We essentially want new ideas in the middle (as we can see runs plateauing). This was partly the reason which drove my RTI-experiments; which are below.

#RTI Runs

To explain the RTIs again:

• Source RTI: Checks whether a domain/paper-inspired proposal is actually faithful to its claimed source.
  • Soft Source is being used here, because strict-source just rejects everything. Another signal of how bad autoresearch is at implementing the domains faithfully.
• Implementation RTI: Checks the code after Claude implements the frozen proposal.
• Proposal RTI: Checks the proposal before implementation. (is it specific or very template driven)

Let’s understand the reasoning for all three RTIs.

Source-RTI:

1. For souce RTI, we wanted autoresearch (claude code) to implement domains/paper faithfully, because we noticed in prior experiments that it tends to struggle.
   1. For example, there was a paper called “SleepGate” which basically talks about how memory in our brains have weights attached to them and that the brain essentially has two layers of memory, conscious and sub-conscious. It explains that the memory travels between these two states and crosses a certain threshold-line. This is where the weights of the memory come into play.
   2. Autoresearch was able to implement the “transfer/travel of memories” but it didn’t capture the main nuance which is that memory have weights.
2. Hence the aim was to make it more rigorous, in our initial testing with strict-source-RTI, the implementations were so bad that it rejected all of them 💀. Hence we had to make a softer-source-RTI, which scores the implementations but doesn’t reject them.
   1. The majority of the scores were low, as the implementations were not faithful, but they weren’t rejected because they did lead to score improvements.
3. This goes back to my initial question of when do we need RTI to kick-in?

Implementation RTI:

1. The aim of this RTI was quite simple; basically to induce a full CLOSED loop.
   1. Claude takes an idea and writes the code -> give the code to codex and ask it to go back to the idea.
   2. If it successfully does, closed-loops are achieved, if it partially does it, continue but make a note of what failed,[4] if it fails, then reject.
2. Codex basically acts a teacher verifying the student’s work by working backwards, if codex is able to derive the original idea, it gives a go-ahead to the claude-code to implement the idea.
3. I found this RTI to be the most useful as it forces a proper implementation and we can also attribute its success to the code-implementation; rather than assumption that it was the code.

Proposal RTI:

1. This was the first RTI i tried; basically the aim was to make the proposals more robust, essentially rejecting any weak proposals before they consume implementation time.
2. To be honest, I find this RTI the least useful, as a lot of initial proposals would get rejected in trying to make them more robust but they would just waste time; as initial implementations/tries are just simple score improvers.

#Discussions

I want to go back to my question of “to maximize efforts or scores on a benchmark, do we need novel ideas AT THE BEGINNING?”

From the RTI interventions which I have tried, lots of things became quite clear:

1. Not everything needs improvement, from the proposal RTI, my intuition changed from “i need this proposal to be robust” to “just get a move on” lmao
   1. To explain in more detail, it was clear that the initial proposals don’t need “robustness” as they are just simple score improvers.
   2. For example in ARC-AGI the first intervention or the first try which the agent does is the same across the three runs because that is the simplest and most explainable score improver.
   3. Because of the proposal RTI, which wanted it to be more robust, it would just double down on a very simple idea and sort of make it more complicated. This score improvement, which the robust proposal would cause, was very negligible.
2. Codex can be biased as well, By this I basically mean that both for the implementation RTI as well as the source RTI, we are completely dependent on codex to understand the idea properly or to understand the code properly.
   1. That dependence on codex can also lead to issues because:
      1. Codex itself could be in an ML attractor basin and it might have its own bias on what works and what doesn't work.
      2. Even though I tried my best for it to be as unbiased as possible by not giving it any clues of the benchmark., that inherent bias will always be there.
3. Do we need “novel ideas” or do we need ideas which work? I think the main question is trying to understand whether we just need novel ideas or do we need ideas which work, because defining a novel idea is very qualitative in nature and it's hard to measure quantitatively.
   1. My sort of answer to this question is basically that we need ideas which are "novel" but also lead to benchmark improvement.
   2. We might need another verifier to verify whether these ideas are novel or not but the main goal is to get ideas which are reversible, “novel” and improve scores on benchmark.

This sort of leads to my next sprint, which is inspired by Aman 's advisor model work. Basically letting RTI become a tool and letting the auto research agent call it as per its need. My reasoning is that: Certain interventions don't need RTI's gate and certain interventions do; so letting it call it as per its need is interesting to explore

Another sort of sub experiment of this adaptive RTI is basically having a shadow RTI which runs in the background, which scores/makes a note of WHEN RTI should have been called by the agent, and we can maybe do RL (only possible on open-source models) with this shadow-rti signal to make them better autoresearch agents. Aman already has discussed some interesting results regarding this, so it would be cool to explore.

---

#Appendix

Notes from review comments on the original doc, with my replies.