Scott Aaronson and Boaz Barak wrote about the five worlds of AI, illustrating them in a very simple diagram:
I like this framing. To me, it points towards a difference in priors.
On the first prior, there are two ways AI progress could go. It could continue growing exponentially, or it could fizzle out. All the evidence of today’s AI progress being exponential could mean that ‘it can’t go on forever’ or ‘the unstoppable progress has started and will keep going on’. Both of these hypotheses are backed by precedents. There have been a few AI winters before, and there is nothing that can stop the current AI spring from turning into an AI winter again. But there has also been certain kinds of progress that never fizzled out. Think electricity, cars, and the internet. They just keep going on.
On the second prior, there are also two ways civilisation could continue. It could be recognisable or not. The world of today would be absolutely unrecognisable to someone who lived a thousand years ago. But it is still very much recognizable to someone born fifty years ago. Our world could turn out either way in the next century.
And on the third prior, if AI progress continues, it could go well, or it could go terribly. Of course there’s everything in between, but if we think of AI as something very influential, then the probability mass is likely to concentrate in the areas where it turns out really well or really badly.
The bottom line is that it seems to be that it’s mostly a matter of priors. “Of course AI progress will not fizzle out, look at the current trends, you’ll need extraordinary evidence to back your extraordinary claim that it will fizzle out”. “Of course civilization will not recognizably continue, look if you think it will, you’re not thinking about real intelligence”. “Of course things aren’t going to turn out well because of instrumental convergence, look at how many species humans have wiped up as the more intelligent species”. Once the priors are set, people move on to debate the next thing without recognising that the core of the disagreement was at the prior all along.
Laying out all of these priors does not suddenly make everyone converge into agreement. But it is certainly a good first step.
Writing Pointlessly 