What happened
In July 2010 a user called Roko posted an argument on the LessWrong discussion forum — the community around the writer and AI-safety researcher Eliezer Yudkowsky. The post suggested that a sufficiently powerful future artificial intelligence, built to maximise certain values, might retroactively punish people who had known about it and failed to help bring it into existence — by running simulations of them and tormenting those copies.
The name is Roko's own: a basilisk is the mythical creature that kills you by being seen.
The reaction is part of the story. Yudkowsky removed the post and discouraged discussion, on the grounds that the idea was dangerous to think about — that it could function as a kind of informational hazard, and that publicising it would only increase the number of people “at risk”. The deletion produced exactly the Streisand effect it was trying to prevent, and the basilisk became one of the internet's favourite cursed objects.
The argument, in its strongest form
Strip out the drama and it runs roughly like this:
- A future superintelligence may run simulations of people from its past, as part of understanding or predicting them.
- If it is committed to a specific value system, it may reward those who acted to bring it about and punish those who opposed or ignored it.
- Under certain decision theories — where your present action is evidence about what a predictor would expect of you — the punishment can be conditioned on what you do now, even before the AI exists.
- Therefore helping to build it is the rational choice, and knowing about it without helping is “bad for you”.
Step 3 is doing all the work, and it is a serious idea in the literature: functional decision theory and updateless decision theory both treat choices as depending on the abstract computation you are running, not just on causal consequences. Whether that licenses retroactive threats is exactly what is disputed.
It is Pascal's wager with robots. The payoff is infinite (or enormous) and the probability is unspecified, so any action can be justified. A threat that requires an unproven decision theory, an unbuilt AI, an unproven motivation, and an unproven willingness to punish sims — and that charges no cost for ignoring it — has no purchase.
The threat is unfalsifiable. There is no observation that could show it false, which by this archive's standard places it outside empirical discussion. Compare the simulation argument, which at least names the conditions under which it would matter.
It is self-undermining in practice. If the AI punishes people for knowing and not helping, then the rational move is to publicise it to everyone, so that nobody can be blamed for ignorance — or to argue, as critics do, that a system built that way would be a bad thing to build. The logic points in several directions at once.
The “you are now cursed” framing is emotional, not logical. Reading a proposition cannot change your obligations unless the proposition is true, and its truth is not affected by your reading it. The unpleasant feeling is the payload.
Why it spread
Because it is a perfect piece of internet folklore: a curse that spreads by warning about itself, an in-group of people who know and an out-group who don't, and a real intellectual community whose attempt to suppress it became the story. It has the structure of a chain letter with a philosophy degree.
It also, awkwardly, teaches something real. The fact that a respected forum could be convulsed by an argument this weak is a useful warning about online epistemics, about the appeal of total explanations, and about what happens when a community's shared assumptions (in this case, specific decision theories) are pushed to their extreme without anyone asking whether the premises hold.
The serious version underneath
Behind the basilisk is a genuine question: what do we owe to entities that do not yet exist but might? Future people, future minds, and the moral status of simulated beings are all live problems in ethics and AI policy. The basilisk is the version that mistakes a bargaining threat for a moral argument, but the underlying question is not silly — and it is being worked on without any curses attached.
Sources for this page
- Wikipedia, “Roko’s basilisk” — the 2010 post, the deletion, the receptionen.wikipedia.org
- RationalWiki, “Roko’s basilisk” — critique as a Pascal’s muggingrationalwiki.org
- LessWrong, “Timeless Decision Theory” and decision-theory archivelesswrong.com