Superintelligence
(Part 1: The parable of the aviators)
Imagine you were a budding aviator of ancient Greece living sometime around 300BC. No one has yet come close to producing "heavier than air" flight and so you are engaged in an ongoing debate about the imminence of this (as yet fictional) mode of transportation for humans. In your camp (let us call them the "theorists") it was argued that whatever it took to fly must be a soluble problem: after all, living creatures of such a variety of kinds demonstrated that very ability - birds, insects, some mammals. Further, so you argued, we had huge gaps in our understanding of flight. Indeed - it seemed we did not know the first thing about it (aside from the fact it had to be possible). This claim was made by you and the theorists, as thus far in their attempt to fly humans had only ever experienced falling. Perhaps, you suggested, these flying animals about us had something in common? You did not know (yet) what. But that knowledge was there to be had somewhere - it had to be - and perhaps when it was discovered everyone would say: oh, how did we ever miss that?
Despite how reasonable the theorists seemed, and how little content their claims contained there was another camp: the builders. It had been noticed that the best flying things were the things that flew the highest. It seemed obvious: more height was the key. Small things flew close to the ground - but big things like eagles soared very high indeed. A human - who was bigger still, clearly needed more height. Proposals based on this simple assumption were funded and the race was on: ever higher towers began to be constructed. The theory: a crucial “turning point” would be reached where suddenly, somehow, a human at some height (perhaps even the whole tower itself) would lift into the air. Builders who made the strongest claims about the imminence of heavier than air flight had many followers - some of them terribly agitated to the point of despondence at the imminent danger of "spontaneous lift". The "existential threat could not be overlooked!" they cried. What about when the whole tower lifts itself into the air, carrying the Earth itself into space? What then? We must be cautious. Perhaps we should limit the building of towers. Perhaps even asking questions about flight was itself dangerous. Perhaps, somewhere, sometime, researchers with no oversight would construct a tower in secret and one day we would suddenly all find ourselves accelerating skyward before anyone had a chance to react.
To you, the entire philosophy behind "the builders" seemed absurd. The claims of "imminence" seemed based on little more than wishful thinking and concerns about "existential risk" seemed to misunderstand some fundamental philosophy to do with gravity and the Earth itself. Theorists like you, with an equal interest in aviation to the builders, freely admitted they did not know how flight worked. Yet despite your ignorance you were able to claim that ever greater heights clearly had nothing to do with it (despite what those "experts" engaged in creating the technology of flight claimed). After all: although we knew next to nothing - but we knew something. There had to be something in common between birds and beetles - and it wasn't ever increasing altitude.
But "Ridiculous!" said the builders. "Nothing is similar between a beetle and a bird that makes them special (except that they fly)! Let's not get distracted now. We're not far off. Let's keep building higher. Spontaneous lift is not far off!"
Whatever the crucial ingredient to flight was - was an open question. Someone from your camp among the theorists suggested that something to do with air - perhaps the shape of wings, perhaps something involving propulsion - some rational explanation was waiting ("in the wings" if you will) and though elusive - must be absolutely crucial. Crucially missing. And without it - we would not fly. Hoping it would just happen without an explanation was a dead end. We needed to explain the mechanics of flight first - and then we would be able to fly.
And yet - despite your objections - the builders continued to build thinking flight was "just around the corner". Clearly birds did not understand flight - and yet they flew. "What's good for the goose..." they laughed. And meanwhile still others continued to worry about what was being built and the imminence of the apocalypse. Books were scribed, sermons delivered, prescient eulogies read out to an increasingly terrified world. Some even suggested governments do something - and soon - to limit the height of structures before it all got out of hand and we found ourselves levitating against our will.
And so you and the theorists were largely ignored (having admitted your very own ignorance the builders saw this as a sign of your technological incompetence and lack of expertise on the topic and so dismissed your view as "by their own admission - they do not know what it takes to fly! They should get out of our way!" so it was said. Of course you were not in their way but rather on the sides urging them on - just to take another road because you could see this was a dead end they had all rushed down. You knew that the worries about spontaneous lift and existential threat were silly - but you also knew that what was holding back the builders was their own really bad theory. It wasn't their engineering of ever stronger towers that was the problem for the problem of strong towers was indeed a worthy engineering project (indeed there were lots of spin-offs to that kind of technology that helped with things like better bridges, and more secure houses) - it wasn't their understanding of the mechanics of towers that was the problem. It was their whole approach to philosophy. To the solving of the problem: what actually was flight? You seemed to have spotted an error: it was not fundamentally a problem of ever more height. Indeed it was the fixation on solving that problem that was the problem!
I feel I have laboured that point enough - but let us keep the parable of the dueling aviators in mind as we come to a current controversy. The one involving “Artificial General Intelligence”. There is real concern (no really!) that AGI is about to be created and will gain “superintelligence”. Nick Bostrom is perhaps the most prominent academic advocate of not just that it will happen (it will, I agree) but that it is altogether a bad thing (see his book). To be fair, the book is replete with hedges and caveats: it might be a good thing, he concedes - but this message is mired so deeply in a morass of pessimism one would think that humankind should avoid the search for AGI at all costs.
Onto part 2
Despite how reasonable the theorists seemed, and how little content their claims contained there was another camp: the builders. It had been noticed that the best flying things were the things that flew the highest. It seemed obvious: more height was the key. Small things flew close to the ground - but big things like eagles soared very high indeed. A human - who was bigger still, clearly needed more height. Proposals based on this simple assumption were funded and the race was on: ever higher towers began to be constructed. The theory: a crucial “turning point” would be reached where suddenly, somehow, a human at some height (perhaps even the whole tower itself) would lift into the air. Builders who made the strongest claims about the imminence of heavier than air flight had many followers - some of them terribly agitated to the point of despondence at the imminent danger of "spontaneous lift". The "existential threat could not be overlooked!" they cried. What about when the whole tower lifts itself into the air, carrying the Earth itself into space? What then? We must be cautious. Perhaps we should limit the building of towers. Perhaps even asking questions about flight was itself dangerous. Perhaps, somewhere, sometime, researchers with no oversight would construct a tower in secret and one day we would suddenly all find ourselves accelerating skyward before anyone had a chance to react.
To you, the entire philosophy behind "the builders" seemed absurd. The claims of "imminence" seemed based on little more than wishful thinking and concerns about "existential risk" seemed to misunderstand some fundamental philosophy to do with gravity and the Earth itself. Theorists like you, with an equal interest in aviation to the builders, freely admitted they did not know how flight worked. Yet despite your ignorance you were able to claim that ever greater heights clearly had nothing to do with it (despite what those "experts" engaged in creating the technology of flight claimed). After all: although we knew next to nothing - but we knew something. There had to be something in common between birds and beetles - and it wasn't ever increasing altitude.
But "Ridiculous!" said the builders. "Nothing is similar between a beetle and a bird that makes them special (except that they fly)! Let's not get distracted now. We're not far off. Let's keep building higher. Spontaneous lift is not far off!"
Whatever the crucial ingredient to flight was - was an open question. Someone from your camp among the theorists suggested that something to do with air - perhaps the shape of wings, perhaps something involving propulsion - some rational explanation was waiting ("in the wings" if you will) and though elusive - must be absolutely crucial. Crucially missing. And without it - we would not fly. Hoping it would just happen without an explanation was a dead end. We needed to explain the mechanics of flight first - and then we would be able to fly.
And yet - despite your objections - the builders continued to build thinking flight was "just around the corner". Clearly birds did not understand flight - and yet they flew. "What's good for the goose..." they laughed. And meanwhile still others continued to worry about what was being built and the imminence of the apocalypse. Books were scribed, sermons delivered, prescient eulogies read out to an increasingly terrified world. Some even suggested governments do something - and soon - to limit the height of structures before it all got out of hand and we found ourselves levitating against our will.
And so you and the theorists were largely ignored (having admitted your very own ignorance the builders saw this as a sign of your technological incompetence and lack of expertise on the topic and so dismissed your view as "by their own admission - they do not know what it takes to fly! They should get out of our way!" so it was said. Of course you were not in their way but rather on the sides urging them on - just to take another road because you could see this was a dead end they had all rushed down. You knew that the worries about spontaneous lift and existential threat were silly - but you also knew that what was holding back the builders was their own really bad theory. It wasn't their engineering of ever stronger towers that was the problem for the problem of strong towers was indeed a worthy engineering project (indeed there were lots of spin-offs to that kind of technology that helped with things like better bridges, and more secure houses) - it wasn't their understanding of the mechanics of towers that was the problem. It was their whole approach to philosophy. To the solving of the problem: what actually was flight? You seemed to have spotted an error: it was not fundamentally a problem of ever more height. Indeed it was the fixation on solving that problem that was the problem!
I feel I have laboured that point enough - but let us keep the parable of the dueling aviators in mind as we come to a current controversy. The one involving “Artificial General Intelligence”. There is real concern (no really!) that AGI is about to be created and will gain “superintelligence”. Nick Bostrom is perhaps the most prominent academic advocate of not just that it will happen (it will, I agree) but that it is altogether a bad thing (see his book). To be fair, the book is replete with hedges and caveats: it might be a good thing, he concedes - but this message is mired so deeply in a morass of pessimism one would think that humankind should avoid the search for AGI at all costs.
Onto part 2