Superintelligence

I have a review in The Guardian of Superintelligence by Nick Bostrom and A Rough Ride to the Future by James Lovelock. I wasn't sure it would work to pair these books, but it seems to have turned out OK as far as it goes.  Here are a few additional comments and notes.

An interesting piece on Roko's Basilisk. "The combination of messianic ambitions, being convinced of your own infallibility, and a lot of cash never works out well."

Bostrom recently outlined his ideas at the RSA. You can listen to the recording here.

Once we begin to celebrate... this phrase is from Thomas Berry's essay The Ecozoic Era. In the western mystical tradition see also, inter alia, Thomas Traherne. A state of awareness that unites elevated cognition and affect might enable what the writer Tim Robinson calls the good step -- though he doubts this is durably achievable for humans: “Can such contradictions be forged into a state of consciousness even fleetingly worthy of its ground?” 

New machines could one day have almost unlimited impact on humanity and the rest of life  See Turing's Cathedral: the Origins of the Digital Universe by George Dyson (2012).

 

killing remotely - already, notes The Economist, America is arguing about whether to give medals to pilotless drones. 

singularity... by around 2030 [discredited] See, for example the resounding meh from Bruce Sterling and this by Alan Winfield. Some analysis suggests consciousness may be intractable to mathematics and the forms of intelligence we identify as most well developed in human societies appear to be dependent on consciousness.

The argument that a superintelligent system will shape the world according to its “preferences” preferences is developed in chapters 5 and 6 of Bostrom's book. The argument that most preferences that such an agent could have will...involve the complete destruction of human life and most plausible human values is developed in chapters 7 and 8.

balance of risks here are the five biggest risks to humanity according to Sandberg et al.

Lovelock thinks...in the very long term...we should welcome-machine-based consciousness.  Sara Imari Walker and Paul Davies speculate that “life forms that ‘go digital’ may be the only systems that survive in the long run and are thus the only remaining product of the processes that led to life.”

For a far out scenario for life in the very very very long term see this.

[superintelligence] will live and experience thousands of times as fast as we can -  here is more from Turing's Cathedral (page 302)

...Organisms that evolve in a digital universe are going to be very different from us. To us, they will appear to be evolving ever faster, but to them, our evolution will appear to have been decelerating at their moment of creation – the way our universe appears to have suddenly begun to cool after the big bang. Ulam's speculations were correct. Our time is become the prototime for something else.

catastrophic risk see It could be worse and this profile by Ross Andersen.

judgement on right or wrong.  Bostrom writes at the beginning of Superintelligence that it is likely that his book is seriously wrong and misleading. He adds, however, that alternative views, including the idea that we can safely ignore the prospect of superintelligence, are more wrong.

There may (or may not) be mileage in thinking about and comparing to scenarios in which superintelligence arrives from outer space. Stephen Hawking is among those who suggest this would probably be a catastrophe for humanity, analagous to the slaughter of indigenous Americans by Europeans. In The Beginning of Infinity (Chapter 9) David Deutsch counters that any civilisation sufficiently advanced to transport itself across interstellar distances would, necessarily, have no need of the raw materials, or anything else, in our solar system. Deutsch continues: “Would we seem like insects to [an advanced alien civilisation]? This can seem plausible only if one forgets that there can only be one type of person: universal explainers and constructors. The idea that there could be beings that are to us as we are to animals is a belief in the supernatural.”

stupidity The first story in Stanislaw Lem's Cyberiad is about a machine which its inventor intends to be fantastically intelligent but which turns out to be incorrigibly stupid. And, of course, in Douglas Adams's Hitchhiker's Guide to the Galaxy, Deep Thought calculates that the answer to the ultimate question of life, the universe and everything is 42. When the receivers of the Ultimate Answer demur, Deep Thought replies that "[he] checked it very thoroughly, and that quite definitely it is the answer. I think the problem, to be quite honest with you is that you've never actually known what the question was."

Image: natural stone arch near Þingvellir in Iceland, site of an early Parliament. Jacob Bronowski warned "we must not perish by the distance between people and government, between people and power."

'Not MY job'

James Lovelock warned at least eight years ago that, collectively, we have hardly begun to get or act together on climate change and global governance. John Beddington seems to agree. And when it comes to what really matters we are still stuck in a giant game of pass the parcel. Three examples this week: Europe shows every sign of procrastination/avoidance of its commitments to poor nations; China blames consuming nations (even though it is using them to grow rich); and Shell pulls out of solar and wind.

George Monbiot wonders what is going on at Shell. But surely there is no mystery. The corporation is following its bliss, aka money. At least its interest in 'new generation' biofuels appears to be honest (and these fuels may even be relatively good news). For decency's sake, however, Shell should cease to pretend that CCS will mitigate emissions from tar sands in the foreseeable future.

Here's to civil society pressure and a heap more technology.

The way ahead

A transition from noncoherent, molecule-to-molecule heat transfer to coherent convection occurs in some heated fluids. During the process more than 1022 molecules come into concert. From a statistical point of view, this is ridiculously improbable. Yet the coherence arises naturally from an applied temperature gradient. Nature creates systems, sometimes quite complex ones, "in order to" get rid of gradients and export atomic chaos into the surroundings. "Centripetal," selflike structures arise from material cycles, energy-driven, self-reinforcing networks. Despite the term selfish genes, genes do not have selves: true selves are cells; without proteins and metabolic networks of recursive amino acids and intermediary molecules, genes are impotent, no more "selfish" than an unplugged toaster.

-- from Into the Cool: Energy Flow, Thermodynamics and Life by Eric D. Schneider and Dorion Sagan.

James Lovelock, who like Lewis Thomas and others, has compared the earth as a whole to a cell, says in his review of Oliver Morton's Eating the Sun:

The wonderful thing about science is that nature itself is always the final arbiter. In time, Gaia theory will be confirmed or denied by evidence from the earth. Unfortunately, we do not have time. The evidence so far suggests that the earth is now in rapid motion towards one of its hot stable greenhouse states, perhaps like that of 55m years ago.

So, what about surviving and thriving? Schneider and Sagan quote Alan Watts (although one might as well quote a proponent of systems theory like Peter Senge, who I think has observed something similar in real life):

A Taoist story tells of an old man who accidentally fell into the river rapids leading to a high and dangerous waterfall. Onlookers feared for his life. Miraculously, he came out alive and unharmed downstream at the bottom of the falls. People asked him how he managed to survive. "I accommodated myself to the water, not the water to me. Without thinking, I allowed myself to be shaped by it. Plunging into the swirl, I came out with the swirl. This is how I survived.

Fine so long as you don't hit your head on a rock. What rocks will there be on the way if/as the Earth switches from one stable state to another, as Lovelock suggests?:

When we change the carbon dioxide content of the air, the earth responds, when healthy, by neutralising our pollution—negative feedback. Now, less healthy, it responds by supplementing our increase with one of its own—positive feedback. The temperature increases rapidly with each addition of CO2 because, over a certain range, temperature and CO2 are directly related and soon the incremental heating from the earth itself will exceed our inputs and then further heating is unstoppable. Fortunately for us, earth history suggests that positive feedback will come to a natural stop and temperatures will stabilise five degrees above the present. The idea that we can stabilise rising temperature at some convenient level, say just two or three degrees above the pre-industrial norm, is probably the delusion of computer modellers. Once positive feedback starts, there may be little that we can do except try our best to adapt to a five-degree hotter earth. Hot enough to make our world a vast desert and starve most of us.

Agricultural practices can obviously adapt to some degree, but only so far.


Gaia, revenge and trust

In an article in today's Independent, James Lovelock says that it is too late to stop catastrophic changes to the global climate. The average temperature will rise by 8 degrees C in temperate regions and 5 degrees in the tropics. Billions of humans of beings will die as a result, he says.

Nevertheless, he says , it is not too late to take meaningful action (however unlikely that action may be): "I cannot see the US or the emerging economies of China and India cutting back in time [emphasis added], and they are the main source of emissions".

Is Lovelock suggesting that action to limit emissions could still limit temperature rises to significantly less than 5 to 8 degrees, or that without action to limit emissions now the rise will be even greater? What is the basis for his conclusion?

When I interviewed Lovelock at length in late 1999 for an article that appeared the following year in edition 21 of Green Futures, his predictions were no less dire than in today's article and in his new book (to be published in Feb 2006).

His thinking has not changed much, then. What has changed is that the science has progressed and has largely confirmed the most pessimistic scenarios of the late 90s. Also, the political situation looks to have deteriorated substantially.

My 1999 article concluded:

Much of what Lovelock has to say is decidedly grim. But there is a positive side to Gaia that he is particularly keen to emphasise. Of at least equal importance to [Gaia theory's] usefulness to science, he thinks, is the [moral] guidance it can offer.

"This has been occupying my attention probably more than anything else. People do need something to revere or worship, and religion is beginning to fade all over the world because it's failing to deliver in two fields. One: it used to be the source of information about life, the cosmos and everything - in other words it did science's job for it. And science now does that job so superbly well that religion has become almost redundant in that sphere. Two: it used to give moral guidance. And it's beginning to fail in that too...And so what do we do instead? Science offers nothing, or hasn't done so far, where moral guidance is concerned".

"But now it just happens, quite by accident and not by any conscious thought on my part or anybody else's, that Gaia does offer moral guidance. It does so because its rules are simple: any species that improves its environment favours the welfare of its progeny, whereas any species that adversely affects the environment dooms it for its progeny. And this is very moral. It gives us something to which we are accountable - the Earth itself".

Gaia, Lovelock stresses, is not and should never be the basis of a religion, because religions have faith. "The word I prefer to faith is trust. If we put trust in Gaia then it gives us something that will fulfil the same kinds of needs as religions have." And the problem is that industrial civilisation in its present form is profoundly betraying that trust.

"I'm a grandfather with eight grandchildren" [says Lovelock]. "I need to be optimistic. I see the world as a living organism of which we are part; not the owner, nor the tenant, nor even a passenger. To exploit such a world on the scale we do is as foolish as it would be to consider our brains supreme and...our other organs expendable. Would we mine our livers for nutrient for some short term benefit?"

Lovelock remains a superb thinker and communicator, whether or not he's right about nuclear power. His reflection in today's article on how Darwin would have responded to Gaia theory is convincing. And the analogy with a boat above Niagra falls (quoted in the accompanying piece by Michael McCarthy) is well chosen.

But it's not clear to me that he's right to talk about total catastrophe as inevitable. There appears to be an inconsistency between on the one hand saying it's too late, and on the other hand refering to changes in future emissions trajectories that could be undertaken in time (however unlikely that change may seem at present).