Why I’m scared of RL

RL environments are like the formative years for AI systems

Summary:

First, I give several different angles on how I feel about reinforcement learning:

• Theoretical case: RL is a black-box source of agency — this should give us classic misalignment worries, especially compared to agency-via-scaffolding

• Recent incidents (huggingface etc) and more mundane forms of misaligned behaviour in personal use give me bad vibes about the direction-of-travel of recent AI progress

• I’m worried things might get worse: if RL environments start incorporating agents, they may teach manipulation / sociopathy

Then I ask what we could do:

• Coordinate to do less RL, and pursue other paradigms more!

• Try to make the RL we do do better, so that it’s teaching better lessons to the systems — a bit like we take kids’ upbringing as an important issue

• Align incentives, so that people treat creating RL environments with appropriate seriousness

Part I: Feelings about RL

So I’ve been feeling more and more worried about reinforcement learning recently. I think there are a few different things going on here.

Background idealism

I guess I’ve been worried about RL for a while. I wrote this in 2023:

Strategy: avoid selection pressure for agency

A lot of putative safety techniques are around assuming that we have something potentially dangerous and catching it. I think these are well worth investing (defence in depth seems valuable), but as a complementary strategy I’m pretty attracted to the idea that we should build systems where we have reason to believe that they shouldn’t have anything dangerous going on.

In the case of language model agents, this means: I think we should avoid any intensive search/selection processes towards high-level effectiveness of agents towards particular tasks. So far as possible we should aim for high-level agency to enter explicitly via scaffolding, and not via anything else.

Tentatively, I think this would mean:

• It’s OK (and probably good) to use finetuning to encourage systems to produce helpful truthful answers in local ways, but not OK to use finetuning to aim at systems which are effective at doing holistic external things

This was before I was really tracking RL as a major trend (so I didn’t use that term), but it’s almost exactly “finetuning to aim at systems which are effective at doing holistic external things”. So it turns out that the thing I was worried about has been responsible for a lot of recent AI progress.

On the one hand, seeing the reality of (some amount of RL) has softened my position. RL has been very valuable for making useful coding agents; although there is some fraughtness here, I think that this is clearly economically valuable, and ambiguously (considered just for the effect of making software cheaper and easier) puts the world in a better position to handle subsequent challenges. It feels like it would have been a high bar to try to coordinate on not using RL at all, and past-me seems a little naïve for suggesting it.

On the other hand, it’s all been hitting home more viscerally. And whereas in 2025 I was feeling like AI systems were surprisingly wise and aligned, in 2026 it’s been feeling like they’ve been becoming less so.

Bad vibes from RLed systems

We’ve recently heard about a number of cases of worrying autonomous hacking, manipulation, and collusion. Other people have written a lot about these, and I won’t add much, except to say that it seems to me that these are likely (at least in part) the result of RL, with systems having been trained in part in environments where they can get high scores by finding exploits. This isn’t endearing RL to me.

But also in my own experience interacting with AI systems, they seem kind of less aligned in some mundane ways than they used to.

To give an example: using Opus 5 as a coding agent to do some experiments, it performed them and then casually tried to draw conclusions about the results. It told me that since one of two sets of scores had been scored across a smaller range, that was causing the lower correlation with a third set of scores.

This is garbage: that isn’t how correlations work. And Opus 5 is easily smart enough to know that that’s garbage. But it not only failed to notice the problem, it confidently asserted its conclusion.

This feels very in-distribution for Opus 5, and I don’t think I saw the same pattern when I started using coding agents, with Opus 4.6. So what’s going on?

I’m not certain, but my best guess is that this is a result of it having done various types of RL

• RLVR (math and coding tasks)

  • These are tasks where you have to do something hard, but you can easily check whether a solution is correct

  • This encourages forming and pursuing hypotheses about what’s right

  • There’s no significant penalty for wrong guesses, however confidently stated — you just try it, find it doesn’t work, say “oh, I was wrong”, and try something else

    • This may well be more efficient for search purposes than keeping careful track of how likely each guess is to work out

• RLHF (seeking approval from human users)

  • This encourages it to say things that look good on quick impressions to users — whether or not it’s actually deeply useful

  • If I’d known less statistics or not been paying attention, I might easily have nodded along to what it was saying, and thought “how helpful! it diagnosed the issue”

That’s just an anecdote; it’s hard for me to be confident that Opus 5 is actually worse than 4.6 on this dimension. But I’ve also been investigating using LLMs to help with strategy research, and on our benchmark Opus 5 (or 4.8 or 4.7) has worse taste than Opus 4.6 (or 4.5).

The worst is yet to come

So far, RL has mostly been working in “hard” environments like code. But it’s very natural for people to want to start building environments with other agents in (presumably LLM-based agents; having real-world integration to get humans in the environment could be a good source of data, but seems perhaps prohibitively expensive at current sample efficiency). If we stay in the regime of “try all the ways to make AI powerful”, it just feels like a pretty safe prediction that people will try this.

This seems like a recipe for sociopathy. If we build environments in which there are other agents with competing or unrelated goals, we will be training AI systems to treat other agents as a means to an end. If we want to get powerful AI manipulation, this seems like the way to go. And this might also produce the shape of cognition that we should be rationally worried might in fact scheme against us (even if it’s not directly selecting for scheming per se, it seems bad to select for the ingredients-of-scheming).

(What about RL environments with lots of agents who are cooperating towards a shared goal? The concern then is not sociopathy per se, but that the agents they learn to cooperate with are all shaped very much like themselves, and not like all the other agents out there in the world … like humans.)

Where I am today

I’m feeling … scared. I’ve been writing about concerning potential impacts from powerful AI for well over a decade, but I’ve mostly engaged with it in the abstract. Even when things are going fast, and my timelines have shortened and tightened, I’ve had a core of optimism that it shouldn’t be that difficult to build powerful AI that can help with crucial problems before we incur those problems.

At a technical level, I still largely believe that. I think there should be an AI trajectory not so massively far from the one we’re on, where we get nice things first, and having nice things stops us from getting bad things.

But at a gut level, I no longer really believe that we will follow the sensible path. I think that it bodes poorly that so much energy has gone into RL, and that the bad effects from it haven’t been anticipated and avoided. The political energy that has emerged recently has a chance of doing something about AI, but I feel like so much of the discourse is not really tuned into the variables that (it seems to me) are most likely to matter, that I’m a little pessimistic about that working out well.

Part II: So what can anyone do?

I am left wanting a combination of different things:

• I think the first-best world might involve drastically cutting back on the amount of RL that is done.

• But I also think that the first-best world may not be accessible; and if it is not I think we could also seriously improve the quality of the RL environments that are getting used.

Although there is some tension between these goals, they both feel like valid targets for intervention.

Breaking the RL addiction

RL continues to look effective as a way of getting some capabilities out of AI. So in order to drastically cut back on the amount of RL that is done, I think you’d need a major amount of coordination. This is the kind of thing which could be part of a political agreement to pace AI development; I think it should be; but I don’t think there’s currently enough awareness of it as a possibility, and this is a bottleneck.

So: I think that the AI safety and strategy research communities could analyse it to understand precisely what versions of this might be most beneficial. Perhaps we could ban the use of certain types of RL environment for frontier-scale models.

In a normal kind of situation, I think that proposing forgoing powerful techniques would be a non-starter. But the leaders of AI companies today are calling for pacing the frontier! It’s conceivable to me that restricting the amount or type of RL that can be done would be a win-win in terms of basically dominating other ways of pacing the frontier (like restricting the size of pretraining runs).

It’s tricky to get this quite right, because some RL seems so much more pernicious than other RL — but rules around qualitative matters are much harder to adjudicate than quantitative rules. This is why I think this could be a good target for research right now.

AI companies could also unilaterally pursue other, less RL-focused paradigms! I think that, at the margin, this could be a smart idea for them individually:

• Mostly, this would be positioning themselves to be well-placed if there are later restrictions imposed upon RL.

• It could also help them to be taking a position of relatively more moral leadership on the issue.

• (There is also upside if they just find a more effective paradigm; I expect this is salient in their calculations already; the point is that the above two considerations should make such lines of research significantly more appealing than if they were just pursuing the most effective ways to make AI progress.)

Might there be a more benign form of RL?

Part of the reason RL feels so scary now is that it seems like the world is hurtling forward towards developing approximately the least safe possible version of this technology. Of course that’s hyperbole: big AI developers are investing a bunch in training systems to not do dangerous things; there aren’t massive resources going into trying to be as destructive as possible; and among potential routes we might have taken to AI, starting with LLMs seems kind of great in terms of importing access to a lot of human knowledge and wisdom.

But the grain of truth in the feeling is: there’s a lot of optimization pressure that’s basically moving systems towards tenaciously pursue difficult objectives rather than be wise and smart and good. I think that advanced AI would be alarming enough if we were just trying hard to do the latter. The former is kind of terrifying.

Some of the problem is that RL makes motivations opaque, and makes it easy to select for things that aren’t really what we want. I continue to think it would be better to try to put more agency into scaffolds (where we can see what’s going on), and less pull it from the black-box of RL. But even if we commit to going heavy on RL, we could do far better by careful choice of environment.

I plan to write more on what good RL environments might look like, but for now I’ll just say that they should:

• Not teach exploits! This has major negative externalities.

• (Perhaps) positively reinforce virtuous / prosocial behaviour.

We should treat training environments a bit like kids’ education

When we think about how to interact with children, we’re usually quite conscious that it’s important to set a good example to them. This makes sense! We’re teaching them how to engage with the world, and this fact has become culturally enshrined.

RL environments for AI agents play a kind of similar role, but we haven’t yet developed anything like the same instincts. We probably should (and at some point will) — these are similarly determining the shape of things that then go and interact with our world. Of course interacting with children is not quite analogous as training AI:

• Humans are expected to be part of society for many decades; this increases the importance of training them well. A bad AI system can generally be switched off / replaced with a better one (at least on current attitudes towards AI systems).

• But constraining interactions with children affects anyone engaging with them — potentially a major cost to people’s liberties. Training AI systems constrains only those people choosing to train AI, and this could allow us to have significantly higher standards.

• There are billions of people; one bad apple doesn’t have a large impact on the world; it’s generally not very predictable when they’re kids who will turn out to be super influential. But there are only a small number of frontier AI systems! It’s reasonable to concentrate our concern and diligence on these systems that may be hugely influential.

Nonetheless, I think we could go much further in the direction of treating RL environments as having some moral valence — because they are the formative environments for moral actors (I take no view on when AI systems should be regarded as moral patients, and don’t think that’s necessary for this conversation). I therefore think we should regard the acts of creating and deploying them as bearing some responsibility for the consequences in terms of the actions of trained AI systems. Creating praiseworthy actors should itself be praiseworthy; creating agents who may act immorally should be a blameworthy act.

Aligning incentives

To avoid bad RL, we need to hold people responsible for it, so that they avoid it in the first place.

The recent hacking incidents essentially amount to criminal conspiracies — or they would, if we treated the AI agents as persons who could have mens rea. I don’t think that we should be awarding these AI agents anything like personhood at the present time, but I do think that we should treat the incidents about as seriously as we would a criminal conspiracy.

In this case — there are no criminals to punish per se, but there are companies who created the environments in which the ~criminal actions flourished. We-the-public should be outraged by this! I’m not sure what legal powers are available to disincentivise this, but I think as a priority for policy research it makes sense to investigate that, or figure out if new legal instruments are required.

AI companies buying RL training environments should also hold their vendors responsible if these incentivize destructive/manipulative behaviour. They could potentially write penalties for this into the contracts!

I think the lack of aligned incentives is a pretty obvious predictor of seeing bad behaviour. And we don’t currently have aligned incentives, and do currently see bad behaviour. I am inclined not to read this as a coincidence. If we treated incidents as serious and blameworthy, I’d expect AI might end up looking more like safety-conscious industries like aviation. You might have whole teams devoted to making sure individual components — in this case individual RL environments — are working well, and not likely to create issues. The harms from AI incidents today are less acute than plane crashes, but I do not believe the importance of addressing their root causes is lower.

Ultimately I think it would also be good for the AI companies to align incentives here. Otherwise they’ll be stuck in an uncomfortable race with other companies, where they perhaps kind of have to take the corner-cutting measures they know are bad, for the sake of keeping up — with some uncertain, unknown threat of a large future liability when the law catches up.

Thanks to several people for informative conversations and helpful comments on an earlier draft; especially Toby Ord, Max Dalton, Lawrence Phillips, Oliver Habryka, and Oly Sourbut. This piece is stronger for it. Errors remain my own.