Minichan

I've seen a lot of people being very skeptical of the idea of Recursive Self-Improvement, which is basically when the AI can update itself/train itself. Supposedly this would lead to a so-called "Intelligence Explosion", which would then create the AI god-machine or whatever. It sounds like another Rationalist fairytale, and Ed, guests, and some people here have taken the stance of "it's not happening, and there's no sign of it ever happening".

I'd like to propose a different theory, which is that RSI is fully possible and feasible, but that it will look a lot less exciting and sexy than it has been sold as, and will have essentially zero impact if it were to happen. It's almost like when Sam Altman made statements suggesting Astra is AGI, but nothing really changed in the world because of it, and it was just another expensive LLM reasoning model release.

My prediction is that Recursive Self-Improvement will go a similar way. If you think about how an AI is trained, it uses a system called RLHF, which at its core involves an LLM generating multiple outputs, a human deciding which one would be best/more appropriate/preferred, and then that influences the likelihood of those kinds of outputs from the LLM in future. As it turns out, OpenAI and other labs farm this part out to contractors and gig workers for various firms. Yes, It's someone's job to sit there and yay or nay various pieces of AI slop to determine which one is 'better'. In some cases, this is Actually Indians, as I've found an Indian tech work finder site (.in domain) giving an overview of the role.

What's also important to know here is that the AI labs are known to be reckless bozos when it comes to AI 'agents'. They're committing felony hacks left and right with these things and setting them up on long-running processes completely unattended. So is it that far-fetched to suggest that they may use these in place of the Indian contractors? I could totally see some engineers working at Anthropic or OpenAI putting an agent running on the previous model in that position, with prompting to "score the messages provided based on XYZ criteria" and letting it spin on that.

That would technically be the AI training itself, and fulfill the definition of RSI at least loosely. However, then you have to sit back and think about that and what it actually does to this process. Does this sound like something that would lead in any way to an Intelligence Explosion? I'm not certain exactly how much these guys are getting paid, but it's got to be far less than the previous-model token spend of running a token-guzzling agent loop to do their job. Also, since it's an AI training the next generation of AI, I feel like that would just lead to another form of Model Collapse, which is when AI gets worse because it was fed the outputs of itself. In this case it's not the dataset but rather the feedback ranking system that's AI-handled, but it just seems like it would have a similar outcome, and that's not even counting the chances of the model hallucinating and doing stupid things with the harness like in the hacks. In other words, RSI would potentially make the AI dumber, not smarter.

In short, don't be surprised if one of the labs drops a press release saying "We've achieved RSI!" or something vaguely hinting at such in the near future, prompting boosters to proclaim how the skeptics are not credible because they 'got it wrong' saying it could never happen, but ultimately nothing ever comes of it and nothing ends up changing about the broken business models powering these companies.