So you’ve taken over the world
Suppose that you were running a big AI project that underwent a fast intelligence explosion. Just weeks ahead of your competitors, you followed the advice of the superhuman strategic AI, and launched cyber attacks to cripple their research in order to avoid an estimated 25% chance of everyone dying. Then with a few months lead, you let your systems steer public and private discourse to give you breathing room for your R&D program to create enough high-effectiveness robots that you can effectively control the world.
You didn’t exactly ask for this. The logic of “for the greater good” led you here. But now — what do you do?
[NOTE: If we end up in this scenario, something has gone horribly wrong; I can’t really imagine plausible circumstances in which I thought it was the right move to pursue. But after the strained setup, it’s a fairly clean thought experiment; it goes in interesting directions; lessons from it may have some generalizability.]
The stakes
At this point, you are the most important person in history. Maybe that’s uncomfortable, maybe you’re into that kind of thing. Whatever.
Honestly what you’ve done so far doesn’t matter so much compared to what you do next. The weight of the cosmos is pressing on your shoulders. Don’t fuck it up. Please?
The challenges
This is a weird situation. In some ways you are in a better position than anyone who’s ever lived. In other ways, not so much.
The difficulties include:
Single point of failure
Right now, if you have a stroke and go crazy, that could be it for the universe
Stress & other cognitive limitations
Getting this right matters tremendously!
There is urgency — both to get out from the single point of failure situation, and because some things may be developing in unhealthy ways in the rest of the world in the interim
You can only think about so much, and may make mistakes (at least for now)
This is all stressful, which makes it harder to think about clearly, which makes it more stressful …
Lack of clarity about how much to trust AI
It’s been fantastic at providing technological advances and strategic advice
It can provide advice on more normative questions (like what to do now), but it’s less clear if the training process means that it should be excellent at that
If AI ratchets to even more superintelligent systems, it’s presently unclear to you whether you should expect to remain in meaningful control
Sycophancy
You can go to other people for advice
But if you have so much power over them, their natural incentives may be very distorted — if they’re scared to give answers you’d be uncomfortable with, that shifts what they may say, and how you can hear it
More broadly, even if people aren’t deliberately distorting their advice, you should be scared that the advice you find seems best (whether it’s from people or from AI) may come apart from the advice that it would be best for you to hear
Power corrupts
It’s an aphorism, but it does seem like having an unusual amount of power, a lack of normal social guardrails, etc., can often mess up people’s thinking
This may be partially a function of the above (e.g. stress, sycophancy), but there could be elements beyond that
Having a vast amount of power could also just make it hard to have relationships with other people that feel normal and comfortable
Principles for responding
Here are some meta-level principles you might aspire to:
Boost wisdom and clear-thinking, in yourself and others
If the world is yours to fuck up (or hopefully not!), then it’s especially valuable to have the kind of thinking that could help to recognise and gracefully steer clear of failure modes
You should want to be wiser yourself, and you should also want access to other sources of wisdom
Empower others, in order to remove distortions in their thinking
If you make sure that some other people have a deep security such that you could not destroy them even if you became angry with them, you may free them to think thoughts that may be important (even if uncomfortable) to hear
Boost AI to provide wise advice
AI might be one of the best sources of advice you have!
Work out how to elicit advice that you can have more trust in (this may be more complex than “have one person ask a single system”); also work out how to automate research to provide advice you’ll feel more robustly good about following
Play it safe with developing more advanced AI
If you’re not very confident that more advanced would be safe, consider carefully how much it’s necessary to rush, and consider going slowly to allow serious exploration of exactly what might make for safe or dangerous further explorations
Create healthy societies with robust epistemic processes
To the extent that you think some societies have been better at surfacing truth than others, work out what the conducive conditions are, and try to replicate those
Broadly, encourage reflection — cognition is now cheap, so try to avoid screwing up for lack of having thought things through
Remove the single point of failure
Hopefully nothing bad will happen to you!
But best not to bet the universe on that — establish structures that are more likely to lead to good things even if you die or go crazy
Note that trying hard to become smarter and wiser looks important; but if this could involve treading new ground, there is always a risk that it leads you somehow off the rails — work out how to make things robust to that
Similarly, if you are horrified by the idea of becoming corrupted with power such that you and people you love would look down on your future self, take steps to guard against that happening
Don’t lock things in prematurely
Except insofar as there is urgency for reasons of the other principles, avoid making long-term commitments. When you have to, try to provide a release valve — some way that if it becomes clear (e.g. to a large majority of people) over time that something was a mistake, the decision is not irreversible
There are some real tensions between these principles! Navigating these will be one of the first challenges you must address.
Some initial actions
Various starting points, with various degrees of urgency:
In the first day
Make sure that you have strategic awareness
What should you maybe be tracking that you aren’t yet? Have AI delegates handle this
Remove the single point of failure
Choose a council of some of the people you most look up to. Pick a quick constitution document for their self-governance, and make sure that they’ll be in charge if you haven’t amended this plan in the next month.
In the first week
Minimal intervention to reduce violent conflict
Sending the right messages, well placed hacking, potentially drone interventions if you’re confident those won’t escalate
Press ahead with automated medical research
People are dying unnecessary deaths every day — let’s move forwards with putting that behind us
In the first month
Guarantee people and organizations important rights
Create a more thought out interim governance plan — the intention should be to gradually move to a mature form of governance that basically everyone sensible feels good about, but not needing to rush key decisions. This is like a better version of the first-day council, with more robustness against you just going crazy.
Communicate with the world!
Be forthright and clear, establishing the start of your relationship with the rest of the world on a foundation that you will feel that you can stand behind with time, and do not regret
Use AI advice to anticipate how people may react hearing it, and which (true) things would be most helpful to include
In the first year
Roll out superhuman AI advice to humanity
Work out what setup would most help you to become the better wiser version of yourself, and start taking steps in that direction
Have serious research programs on:
Working out how to advance AI further, safely
A final word
Of course I don’t know what I’m talking about. I’m writing before any of this happens, having given it vastly less thought than you have available at your fingertips. You should take everything I say with a generous serving of salt.
Why listen to me at all? Perhaps because I don’t know what I’m talking about. I don’t know who you are or what’s been happening; so I’m unusually well insulated from the political pressures du jour. I’m writing as a thought experiment for people in my time — trying to address the puzzle of what would actually be good for someone in your situation to do — as much or more than I’m writing to you.
If I had to distill down to two core challenges you face, they would be these:
Rushing to give up power risks making big mistakes that you (and many others!) may later regret. But having seized power has already harmed people, and each day you hold onto it compounds that harm.
You need wisdom so much that it becomes harder to access. AI systems are powerful but not necessarily trustworthy on these normative questions. Everyone around you has distorted incentives — and so do you. It may be uncomfortable even to think about the ways in which you’ve hurt people (even if you were correct that there was no better choice — and even more so if you might have misjudged things somewhere along the way), but that stands in the way of thinking sensibly about what to do next.
People have always wrestled with versions of these challenges. Monarchs, political leaders, CEOs, philanthropists. Many of them have handled them poorly, perhaps without even letting themselves see how they are. But that was amateur stuff; you’re playing for real stakes.
Good luck.