Superintelligence

Paths, Dangers, Strategies

Nick Bostrom

Summary: How shall we survive in the world where we are not the most intelligent species?

Score: 75 / 100

My Goodreads review

Very important book. I like the nerdy approach, but it was quite hard to read about the possible futures of AI progress. I only skimmed last 3 chapters.

1 chapter – history

I believe that after achieving human level AI, it wilĺ be a matter of a few months to achieve superhuman level. It is enough to throw more cpus, gpus, just more hardware used for the HLAI.

I believe that AI will be useful before reaching AGI, for formal probems, material invention, solving theorems, finding patterns in data, not necessary to have consciousness for these tasky.

2 chapter – paths to SI

Z čeho se vude AI učit? Rozhodne nemuže z internetu, ktery je zabalasteny blboatmi, musela by je oberovat ale i v clancich a na wikimjsou mepresnosti…

AI asi bude muset bootstrapovat znalosti. Leda by sama vytvorila superAI ktera bude operovat na kvalitativme jine urovni, napr delat zakladni vyzkum pomoci inteospekce apod.

verim, ze hardware uz je schopny minimalne human level AGI

2022-08-02

Letadlo je 100,000x jednodussi nez ptak. Stejne tak AI muze byt 100000x jednodussi nez mozek. Neni duvod, aby system AI mel byt extremne komplexni, nepotrebujeme modelovat neurotransmitery…

2022-08-24

I hope that a future superintelligence won’t read this book. It would instantly know how to escape from its confinement.

Isn’t the argument for the paperclip disaster flawed? Wouldn’t be the superintelligence intelligent enough to not stupidly and literary obey the task of making more and more paperclips?

2022-10-07

V šalině se proklikám od 12. kapitoly na konec na závěr. Myšlenka: vynález je přenesení myšlenky z budoucnosti do současnosti. Kdyby se věc nevynalezla, přišlo by se na ni později.

Clippings

Only Scronkfinkle, a one-eyed sparrow with a fretful

AI pioneers for the most part did not countenance

our standards for what is impressive keep adapting to the advances being made.

John McCarthy, who lamented: “As soon as it works, no one calls it AI anymore.”

Donald Knuth was struck that “AI has by now succeeded in doing essentially everything that requires ‘thinking’ but has failed to do most of what people and animals do ‘without thinking’.

recover most of the losses. But for a while, at the trough

Success at emulating a tiny brain, such as that of C. elegans, would give us a better view of what it would take to emulate larger brains.

By contrast, it seems likely that somebody could in principle sit down and code a seed AI on an ordinary present-day personal computer; and it is conceivable—though unlikely—that somebody somewhere will get the right insight for how to do this in the near future.

superimposing photographs of sixteen different individuals (residents of Tel Aviv).

there is no reason to suppose Homo sapiens to have reached the apex of cognitive effectiveness attainable in a biological system.

we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it. Progress

we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.

the human retina transmit data at an impressive rate of nearly 10 million bits per second,

We do not need to plug a fiber optic cable into our brains in order to access the Internet.

Self-deception detectors might be even more powerful.

And is it not the case that contemporary humanity is idolizing material consumption, depleting natural resources, polluting the environment, decimating species diversity, all the while failing to remedy screaming global injustices and neglecting paramount humanistic or spiritual values?

(Anything the brain does in under a second cannot use much more than a hundred sequential operations—perhaps only a few dozen.)

Axons carry action potentials at speeds of 120 m/s or less,

The process of solving a jigsaw puzzle starts out simple—it is easy to find the corners and the edges. Then recalcitrance goes up as subsequent pieces are harder to fit. But as the puzzle nears completion, the search space collapses and the process gets easier again.

Human decision makers often seem to be acting out an identity or a social role rather than seeking to maximize the achievement of some particular objective.

The principal reason for humanity’s dominant position on Earth is that our brains have a slightly expanded set of faculties compared with other animals.

von Neumann probes,

Imagine their brains lying side by side in quiet repose.

We humans often seem happy to let our final values drift. This might often be because we do not know precisely what they are.

Bertrand Russell, who spent many years working on the foundations of mathematics, once remarked that “everything is vague to a degree you do not realize till you have tried to make it precise.”24

of total global income is received as rent by owners of capital, the remaining 70% being received as wages by workers. If

30% of total global income is received as rent by owners of capital, the remaining 70% being received as wages by workers. If