scrape_r_rational/data/cache2/7a0suv.md

Eliezer Yudkowsky is writing a new book

• Author: u/monkyyy0 *

• URL: https://equilibriabook.com/

• Score: 63

• Created: 2017-11-01T02:25:38

Post:

Link to content

Comments:

u/trekie140 [+14] (10 hours later)

I really like this idea and will definitely check this book out, but I’m not setting my expectations too high since I rarely found his blog posts more insightful than articles written by economists. Not that I think economic analysis is an exact science, I’ve just studied it enough myself that I trust it as one of the lenses to view social structures through.

u/None [+8] (a day later)

I liked his early LW Stuff more than I would like a blog post by most other people. (With possible exception of Scott Alexander).

That said I really hope he commissions an external editor and/or feedback from outside "the community." A lot of his stuff that could have been really great has been hampered by not being accessible to a broad audience

u/FeepingCreature [+7] GCV Literally The Entire Culture (11 hours later)

Does anyone know a good German translation of "adequacy"? I'm tempted to go with "competence" as a stand-in, not least because "Inkompetenzanalyse" is snappy and directly understood.

Edit: Zulänglichkeit?

u/RoggBiv [+2] (18 days later)

What about Adäquatheit? That's basically the literal translation.

Source: am German, but there might be a better word for it still

u/FeepingCreature [+2] GCV Literally The Entire Culture (18 days later)

Hab ich auf den ersten Blick erstmal als "Aquadiät" gelesen. :) Würde schon gehen, aber ich glaub Zulänglichkeit fließt besser.

Ich hab meine Übersetzung auch erstmal liegen gelassen, in der Hoffnung dass Eliezer erstmal einen guten Redakteur findet.

u/Anderkent [+1] (4 days later)

hinreichend? Am not very good at german.

u/monkyyy0 [+4] (3 minutes later)

It sound to me that he's going to start advocating for agorism or something like it but more general; lots of game theory suggesting the existing system can't improve and he's not the type to just say "can't fix it" and his example is clearly working outside the system, a system that well respected to the general population.

u/EliezerYudkowsky [+26] Godric Gryffindor (an hour later)

Nope.

u/monkyyy0 [+11] (an hour later)

Hmmmm, where are you headed in this then?

It is non-fiction spoilers are ok

u/KJ6BWB [+7] (12 hours later)

It is non-fiction

Dang it.

u/AntiTwister [+5] (6 hours later)

lots of game theory suggesting the existing system can't improve isn't optimizing what you want it to optimize

The impression I get is that this is about better identifying which sorts of optimizations are incentivized or impeded by large scale systems. This allows you to form better priors about the value of reasoning from first principles in a particular domain, and whether this is likely to yield an improvement in your optimization criteria over whatever group consensus is currently systemically supported.

u/monkyyy0 [+3] (6 hours later)

¯_(ツ)_/¯

I may be projecting a little. I don't see "hope the people in charge are sane" as a solution, and hope that's not where he's going with this. Isn't this sort of thing already apparent to people inside the system? If knowledge of a problem was enough I would think even blind attempts to fix a poorly defined problem would eventually help.

u/LupoCani [+13] (6 hours later)

¯_(ツ)_/¯

You have to write ¯\\\_(ツ)_/¯ for it to show as ¯\_(ツ)_/¯. The first backslash escapes the second backslash, making it visible, and the third backslash escapes the underscores, which would otherwise just create italics around the face part:

Input	Output
¯\_(ツ)_/¯	¯_(ツ)_/¯
¯\\_(ツ)_/¯	¯\(ツ)/¯
¯\\\_(ツ)_/¯	¯\_(ツ)_/¯

u/monkyyy0 [+16] (7 hours later)

¯_(ツ)_/¯\

u/NNOTM [+6] (15 hours later)

actually, you should write ¯\\\_(ツ)\_/¯. Otherwise, look what happens when you have italic text after the kaomoji:

¯\_(ツ)_/¯ is shrugging.

Now with the extra backslash:

¯\_(ツ)_/¯ is shrugging.

u/LupoCani [+6] (17 hours later)

Alright, that is one devious special case. Good to know.

u/ciphergoth [+8] (18 hours later)

And to write ¯\\\_(ツ)\_/¯ you have to write ¯\\\\\\\_(ツ)\\\_/¯. And to write that I had to write... never mind.

u/LupoCani [+3] (a day later)

That's why you use *code tags* to get away with however many \backslashes\ you desire.

u/VanPeer [+5] The shard made me do it (a day later)

Thanks for writing this book. I really like the questions being framed in the first two chapters, which is one of the biggest incentives for me in reading your work. BTW, do you benefit more if I pre-order from MIRI instead of Amazon for the same price? Sorry if this has been asked before.

u/EliezerYudkowsky [+5] Godric Gryffindor (2 days later)

I actually have no idea, in this case. My guess is we'd mainly want you to read the book in whatever format is most convenient to you.

u/VanPeer [+2] The shard made me do it (2 days later)

Ok, just pre-ordered the MIRI download to incentivize you a bit more to finish the book (if it isn't already).

u/Tenobrus [+3] (16 hours later)

Quick question: the website says printed copies are available to pre-order off Amazon, but it looks like only the Kindle version is available at the moment. Are printed editions not actually planned/is the site in error?

u/wren42 [+3] (47 minutes later)

self sustaining mini agricultural communities are probably the only way most of the population survives the coming crash/techno revolution.

u/monkyyy0 [+24] (an hour later)

I think it may be a little early to predict the downfall of civ, we are still breaking even

u/wren42 [+7] (9 hours later)

How exactly are we breaking even? We are over consuming and the vast majority still live in poverty.

u/monkyyy0 [+12] (10 hours later)

I believe the number of poeple eating enough is still increasing, while quality of life in the west is somewhat stagent that's a far cry away from radical unrecoverable decline.

Crime rates are dropping golbally and while the supply chains for tech are of questionable stablity(percentage of electronics that touch Shenzhen are a little worrying) it's not like we couldn't restart.

Things look bad compared to what we want not to what they were.

u/wren42 [+6] (12 hours later)

There are three major factors I'm considering when predicting future collapse (unless major changes to our civilization occur)

1. Climate Change

This is a big problem. Pretty much all scientists agree major problems are in store, yet governments and industry are not responding quickly enough. There will be disruption to our food supply and environmental stability unless drastic changes occur, which appears unlikely.

2. Resource consumption

The world is already over consuming both renewable and non-renewable natural resources. As india and china continue to industrialize, this is expected to increase.

http://www.overshootday.org/

https://www.ecowatch.com/humans-consumption-of-earths-natural-resources-tripled-in-40-years-1943126747.html

Global consumption behaviors will have to change dramatically to reach equalibrium. With population continuing to rise, this means restricted access to goods overall.

3. Economic inequality and the AI revolution

It doesn't matter if we are wealthy if most people don't have access to that wealth. Inequality in wealth distribution is growing dramatically. The most likely outcome of the coming AI revolution is a dramatic shift in power away from the labor class and toward the capital class. When automation reaches the remaining blue and white collar jobs and human labor is devalued, the current economic system of producer-consumer breaks down. The capital class will for the first time not need the masses at all. Without policy intervention, the outcome of this is widespread unemployment and poverty. Surviving this for most people, unless some sort of UBI system is established, will come down to building small self-sustained agro communities, as I suggested above.

u/entropizer [+10] (18 hours later)

This is a big problem. Pretty much all scientists agree major problems are in store, yet governments and industry are not responding quickly enough. There will be disruption to our food supply and environmental stability unless drastic changes occur, which appears unlikely.

Climate change is going to cause migration and starvation.

2. Resource consumption

The world is already over consuming both renewable and non-renewable natural resources. As india and china continue to industrialize, this is expected to increase.

http://www.overshootday.org/

https://www.ecowatch.com/humans-consumption-of-earths-natural-resources-tripled-in-40-years-1943126747.html

Global consumption behaviors will have to change dramatically to reach equalibrium. With population continuing to rise, this means restricted access to goods overall.

Some things are going to be more expensive.

Inequality in wealth distribution is growing dramatically.

Currently false.

I agree the things you talk about are problems or potential problems, but we're not going to reach a stage where most of the remaining humans live in small self-sustained agro communities unless there are some major pandemics or nuclear wars.

u/ben_oni [+2] (18 hours later)

1. Climage Change

This is a big problem

Unlikely. Changes are likely to be slow enough for the economies to adapt to in real time. My priors suggest that models saying otherwise are bogus.

2. Resource consumption

You do realize we're currently trying to bootstrap a sustainable economy by exploiting the existing non-resources, right? This is something that always needed to happen. As non-renewable resources are exhausted, the economy will naturally shift to using the now-cheaper renewable options.

3. Economic inequality and the AI revolution

the outcome of this is widespread unemployment and poverty

Unlikely. Much has been written on this topic. What happened to all the weavers and bakers and blacksmiths? The fact that blue-collar workers in first-world countries use technology on a continual basis kinda means what it means to be part of the "masses" has changed dramatically. Expect more of the same in the future.

As you say, and I agree, the most likely outcome will be a guaranteed minimum income. Since we already have various welfare and social security programs, replacing these programs with a UBI will be pretty straightforward. This is already being discussed at all levels. I think it's still a bit early, but we'll see. This will probably be paired with various forms of population control.

The real trick will be to modernize those parts of the world that are resistant to modernization.

u/wren42 [+3] (23 hours later)

Lots to respond to, but first things first: what are your sources on climate change and calling runaway scenarios "bogus"?

u/ben_oni [+1] (a day later)

what are your sources on climate change and calling runaway scenarios "bogus"?

Too many models have been wrong. Not being a climate scientist, I have to take the predictions and see if they come to pass. Every time they don't, my confidence in the next decreases.

Also, controls theory. Take the temperature data, remove the labels, show it to a controls engineer, and ask them what it looks like. I've done this, and the answer I received was "ringing around a new setpoint".

And finally, doom-mongering. Doomsayers have been predicting the end of the world since the beginning of the world. They always have reasons, and they sometimes sound reasonable. Sometimes it actually happens (when reducing the scale of "the world" to something more localized). Climate scientists that advocate for policies are not doing science: they are doomsayers.

So yes, I have lots of reasons to doubt the runaway scenarios. Gasoline is not $15/gallon, California, New York, and Florida have not fallen into the sea, and the polar bears are not extinct. And I have no reason to believe that the cause for the previous outlandish predictions has gone away.

u/wren42 [+3] (a day later)

so, no actual science then. your own back of the napkin work and opinions. doesn't sound super rational, to be honest.

actuals vs predictions: http://www.climatecentral.org/news/ipcc-predictions-then-versus-now-15340

u/ben_oni [+3] (a day later)

So, no back-of-the-envelope calculations. Just outright lies. Fine. You know, you could just read the actual reports.

Besides, what could be more truly science and rational than using your own observations? Try it sometime.

u/Izeinwinter [+2] (a day later)

Climate change will not kill the first world because the first world currently spends so laughably low a percentage of its productive output on food production that shifting to a basis of "Storm-proofed greenhouses, so many armored-up greenhouses" would do bugger-all to the overall economic sustainability. That scene from bladerunner 2047? Like that.

u/wren42 [+2] (a day later)

It's fine if the environment's fucked, we'll all just live inside.

u/Izeinwinter [+3] (a day later)

Its not fine. But the specific scenario I see a lot "And then everyone starves to death", basically cannot happen, because it presumes no response to a deteriorating situation at all, and learned helplessness is not that powerful. You would need a "and then, global thermo-nuclear war" in there somewhere.

u/wren42 [+2] (a day later)

Brazil.

u/sparr [+7] (2 hours later)

we are still breaking even

Are you ignoring pollution and sea level rise?

u/monkyyy0 [+10] (3 hours later)

Yep.

u/LazarusRises [+2] (a day later)

Why?

u/monkyyy0 [+2] (a day later)

A one degree temperature difference will matter to a bunch of frogs; humans on the other hand can handle every environment.

u/LazarusRises [+6] (a day later)

Also matters very much to a lot of sea organisms, many of which are carefully calibrated to the pH of the ocean, which gets lower as the water gets warmer. Both those sea creatures and frogs are part of delicate and complex ecologies, the collapse of which would mean starvation for at least many developing nations.

u/None [+2] (a day later)

No. But we also shouldn't ignore the potential solutions.

u/sparr [+2] (a day later)

potential

I don't think you understand what "breaking even" means.

u/GaBeRockKing [+0] Horizon Breach: http://archiveofourown.org/works/6785857 (an hour later)

most of the population survives the coming crash/techno revolution.

Nah. The overwhelmingly vast proportion of the population will be emulated minds because it'll be far cheaper to spin up EMs that it would be to make a biological person. Compared to the preponderance of people living in digital space, only a minority of people would die in any techno revolution.

u/ben_oni [+6] (2 hours later)

You are seriously over-estimating the potential of computation and emulation.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (3 hours later)

Nah. Human brains are extremely unlikely to be the most compact possible storage medium. And unlike humans, EMs wouldn't need all the external stuff we need to keep our bodies running like food and water.

u/wren42 [+4] (9 hours later)

The majority of the future population maybe. I'm talking about current pre transition population, most of whom are just fucked.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (15 hours later)

I'm talking about current pre transition population, most of whom are just fucked.

Well yeah, but by the time the techno-revolution gets to the point where it's murderizing meatspace, we're morally negligible anyways.

u/wren42 [+6] (16 hours later)

I'm not talking about intentional destruction of humans by some robot army.

I'm talking about the much slower but just as devistating process of economic decline and widespread poverty that will accompany the automation of larger and larger % of the workforce, in conjunction with the further concentration of wealth into the hands of the super rich.

u/GaBeRockKing [+2] Horizon Breach: http://archiveofourown.org/works/6785857 (16 hours later)

The point where the automation gets really bad is when computers are smarter than people. Incidentally, that's also the point where computers are people, and therefore of equivalent moral weight.

u/wren42 [+5] (16 hours later)

Mmm I disagree on two counts.

First, automation will be socially disruptive before string superhuman ai. There are degrees of separation there.

Second intelligence -or in this case effectiveness at performing a task- is not the same as having personhood or moral weight. Further, this doesn't DEvalue human moral weight. Adding more people doesn't make many dying ok.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (17 hours later)

First, automation will be socially disruptive before string superhuman ai. There are degrees of separation there.

I have no doubt it's going to be disruptive, I just very much doubt it'll be "kill 3.5+ billion people" disruptive. Or rather, "kill significantly more than 3.5 billion people" disruptive, counting replacement rates. At least, that is, before we get fully sapient machines.

Second intelligence -or in this case effectiveness at performing a task- is not the same as having personhood or moral weight. Further, this doesn't DEvalue human moral weight. Adding more people doesn't make many dying ok.

The point where you can run human-level AI is the point where you can scan human brains (if not necessarily at high fidelity) and emulate them. Regardless of how expensive such an operation would be, it only needs to happen once to get a whole bunch of distinct EMs so long as the EMs, after being spun up, get put in different situations.

Further, this doesn't DEvalue human moral weight. Adding more people doesn't make many dying ok.

It devalues human moral weight as a proportion of the total moral weight of living things. It the utilitarian premise that the needs of the many outweigh the good of the few. You can disagree with utilitarianism, of course, but that doesn't stop the majority from acting self-interestedly.

u/wren42 [+1] (23 hours later)

No, you don't need super human ai and brain scanning technology for major economic disruption to occur. We will see machine intelligence capable of replacing the majority of human labor long before we get true super ai or Em's.we are seeing it already.

To your last paragraph, if your response to "millions of people suffer and die" is "oh well, we can make more in our computers", your utility function fucked up somewhere. I sincerely hope no one like you is in charge of any important decisions in the coming century.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

No, you don't need super human ai and brain scanning technology for major economic disruption to occur. We will see machine intelligence capable of replacing the majority of human labor long before we get true super ai or Em's.we are seeing it already.

But we do need it before we get your apocalyptic end of the world scenario where everyone's killed off.

To your last paragraph, if your response to "millions of people suffer and die" is "oh well, we can make more in our computers", your utility function fucked up somewhere. I sincerely hope no one like you is in charge of any important decisions in the coming century.

Quite the opposite. Why should my utility function value the lesser group over the greater group?

u/wren42 [+1] (a day later)

You are thinking too black and white. There will be suffering long before any "apocalypse where everyone is killed off". Would you personally like lose your job, home, access to the internet, and transportation? Could you survive? Maybe. Is it a good outcome? No.

And your ethics are very messed up. The fact that you could create artificial minds at some future point doesn't alleviate or invalidate current suffering. It doesn't excuse it in any way, especially if they aren't even causally connected. This isn't a trolly problem where some must suffer or die for others to exist. This is like a trolly problem where good policy could stop the trolley, but instead you just suggest we create more people on one side so the others become inconsequential. It's madness.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

You fundamentally misunderstand my point. I'm not saying suffering is good, or even necessary, I'm saying that the original poster is seriously overestimating the long-term importance of meat humans.

u/wren42 [+2] (a day later)

And I'm saying the possibility of future Ems has Zero bearing on the current value of living people. It's irrelevant. Yet you raised it as some sort of counter argument, as if the solution to "lots of people suffer and die" is "instantiate more artificial minds."

Honestly I think your value system is just fucked.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

And I'm saying the possibility of future Ems has Zero bearing on the current value of living people. It's irrelevant. Yet you raised it as some sort of counter argument, as if the solution to "lots of people suffer and die" is "instantiate more artificial minds."

Luckily, I'm talking about a time where EMs will be the preponderance of living people. Your value system is fucked up if you think the lives of mere billions will outweight the lives of trillions or even quadrillions.

u/wren42 [+2] (a day later)

so you are talking about something completely irrelivent to the point?

We were discussing the problem of economic decline causing suffering for lots of people. for some reason you keep indicating this isn't a problem, and you keep bringing up EMs. Are you just changing the subject, or is there a relationship?

u/GaBeRockKing [+0] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

We were discussing the problem of economic decline causing suffering for lots of people. for some reason you keep indicating this isn't a problem, and you keep bringing up EMs. Are you just changing the subject, or is there a relationship?

You evidently were, but I wasn't. I was countering the thread op's premise that the only way to survive the majority of people dying in a technological revolution would be through substinence agriculture. My position is that his point was a nonstarter since a technological revolution capable of such widespread murder would also have the majority of people as EMs anyways.

u/wren42 [+1] (a day later)

I'm OP.

It has nothing to do with murder.

It has to do with economic dispossession. No one is talking about murderbots. We are talking about the impacts of the movement toward full automation on anyone who doesn't own an automated factory.

unless there's UBI, the answer is "most people are SOL", and left to fend for themselves. So, they form communes, or they starve.

u/GaBeRockKing [+0] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

In that case, my response is simple. Namely, that "most" people won't need to do either of those things, because most people will be EMs. That's not to say the moral weight of those people will be zero, just that the moral weight of those people will pale in comparison to the moral weight of the EMs.

u/wren42 [+1] (a day later)

so you have no response. your just continuing to ignore the topic and repeat the same unrelated thing.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

Bullshit. You said my utility function was messed up. I countered. You said the majority of people would be reduced to substinence agriculture. I countered that too. What more do you want? You are deliberately misinterpreting my statements because you have some postapocalyptic agrarian fantasy you want to stick with.

u/wren42 [+1] (a day later)

no, you are randomly changing the subject! You flip flop between saying you aren't even talking about the same topic, and then saying Ems are your answer to poverty.

My assertions:

Automation will lead to widespread loss of quality of life unless UBI is implemented.

Your response: EMs will exist someday so who cares?

If you have some OTHER response to the problem of poverty caused by the loss of value for human labor, lets hear it.

If your only response is that EMs might exist some day so poverty doesn't matter, then yes, your utility function is fucked up.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

Automation will lead to widespread loss of quality of life unless UBI is implemented.

This is absolutely not what you said.

self sustaining mini agricultural communities are probably the only way most of the population survives the coming crash/techno revolution.

That was your first post, which I responded to.

But with reference to your new argument,

Automation will lead to widespread loss of quality of life unless UBI is implemented.

My counter is that, as a proportion of a population, regular humans will be negligible compared to EMs and AIs. Therefore it would be un-utilitarian to consider their preferences over the preferences of EMs and AIs. So because energy is a finite, nonexcludable resource, and since, per person, EMs will require less energy than regular humans, the only moral option is to favor EMs, as opposed to regular humans, because resources allocated towards the benefit of EMs result in more preferences being satisfied than resources allocated towards the benefits of humans.

With regards to your first post, I was railing against the statement "most of the population" you included in your first post, because it implies (incorrectly) that the regular humans are going to be the part of the population that matters.

It's not a fun thing to realize, considering we are regular humans, but using a general utilitarian argument, when utilitarianism would judge against you, just makes you fish in a barrel.

Of course, any deontological view you have about the issue (ex. EMs aren't morally relevant because I define "morally relevant" as only including humans) remains correct internally, but you'll have a hard time getting the EMs to agree with it.

u/wren42 [+2] (a day later)

What did you read some Robin hanson and have your mind blown or something? You are obsessed with derailing on this hokey EM thing.

If you insist, honestly that future is wildly improbable. Why should a GAI controlled by a technocracy bother emulating billions of people at all? There's 0 incentive for that scenario to even exist.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

You are obsessed with derailing on this hokey EM thing.

You're the one who started talking about some post-doomsday agrarian society. I just made a more likely prediction. Maybe it isn't the most likely, but it's certaintly more likely than what you proposed.

If you insist, honestly that future is wildly improbable. Why should a GAI controlled by a technocracy bother emulating billions of people at all? There's 0 incentive for that scenario to even exist.

If there's going to be a GAI that takes total control, neither UBI or agrarian communes are going to stop it from paperclipping the planet.

u/wren42 [+1] (a day later)

Are you suggesting we'd have EM without GAI? because that's just plain silly.

there is literally nothing likely about a society with billions of EMs. It's a tremendous waste of resources for no purpose whatsoever.

and obviously you could have GAI without a paperclip scenario.

regardless, you still seem to be having trouble with the idea that the future can be made up of more than one moment in time. UBI is important PRIOR to a full AI singularity, but DURING the period while weak AI automation is playing havoc with the economy -- ie. soon.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

Are you suggesting we'd have EM without GAI? because that's just plain silly.

Having EMs, even low fidelity ones, is a prerequisite for having GAIs. We won't figure out how to make general intelligences without understanding the most general intelligences we've found yet.

It's a tremendous waste of resources for no purpose whatsoever.

The exact same could be said of our society. We don't need humans, there is no inherent value to a human life assigned by the universe. But the thing is, humans assign value to their own existence, which is why we keep propagating. EMs will be the same way-- some people will want EMs of themselves, and EMs will be able to do the same cerebral tasks as humans, so EMs will propagate indefinitely (bounded by hardware, of course.)

UBI is important PRIOR to a full AI singularity, but DURING the period while weak AI automation is playing havoc with the economy -- ie. soon.

self sustaining mini agricultural communities are probably the only way most of the population survives the coming crash/techno revolution.

So it looks like you've narrowed the scope of your argument significantly since your first post. And to that I say... OK. I'm not here to argue about near-term social safety net policy. If you're defining the scope of your argument to within the period before any significant advances towards GAI, then my argument doesn't apply because I'm talking specifically about the latter end of the process, where AI become strictly better than humans at the majority of tasks.

Maybe UBI is the best option in the short term, maybe it isn't.

u/ben_oni [+1] (21 hours later)

I'm talking about the much slower but just as devistating process of economic decline and widespread poverty that will accompany the automation of larger and larger % of the workforce, in conjunction with the further concentration of wealth into the hands of the super rich.

Thanks Karl.

It's absurd to argue a point that has already been proven false. The history of the twentieth century demonstrates the opposite. You can argue that the particulars of the future are different, but then you need to look at specific reasons instead of making generalized assertions.

u/wren42 [+1] (23 hours later)

We aren't talking about what has already happened, though there's plenty of president for suffering of the poor during economic disruption. I am talking about a specific trend we can see presently. Within our lifetimes general doctors, lawyers, accountants, simple IT jobs, and many other white collar careers will slowly disappear. Once we get to AI that can effectively outperform humans labor loses its value. This process started with the industrial revolution yes, and there's been a period of recovery and growth, to but that doesn't mean it's over or that the trend of automation is a fiction.

u/ben_oni [+1] (a day later)

I'm confused. You don't see how periods of economic disruption are followed by periods of recovery? Yes, automation disrupts economies and industries. But the people are fine. So what if there are no accountants in 30 years. Does that mean all the accountants will be in abject poverty? Not likely.

It sounds to me like your saying that even though this cycle has happened before, for some reason this time it will be different. That sounds suspicious to me. You sound like a Marxist.

u/wren42 [+1] (a day later)

But the people are fine.

It looks that way to the winners after the fact, but ask the actual workers during the industrial revolution, or farmers during the dustbowl if they were fine.

The vast majority of people in the world DO live in poverty. So many Western "rationalists" have this glaring bias of prosperity due to having been raised in luxury, failing to see that their entire civilization is based on the brutal exploitation of cheap labor in other societies.

Would you be ok living like a factory worker in China? Because that's a real world scenario. Middle class western suburbia is the fantasy, and it's eroding fast.

once software comes for the most common white collar jobs, the west is going to have a rude awakening. Suddenly we are going to have to live like the rest of the world, the sorry sots we've been looking down on and telling ourselves would be fine if they just got their shit together.

And yes, the cycle WILL be different -- because when you start talking AI and full automation, you aren't just changing the relationship between labor and capital, you are breaking it. A fully automated industry doesn't need labor at all. the market for it disappears. this is an entirely new animal, and most people aren't going to be fine.

u/ben_oni [+0] (a day later)

The vast majority of people in the world DO live in poverty.

You have no idea, do you? You're so wrapped up in your fantasy you can't see the obvious. Why not just ask them?! Ask that factory worker in China how things are going. Compare apples to apples. How is he relative to his counterpart a hundred years ago? That's the real comparison.

once software comes for the most common white collar jobs

They are releasing a new software system that will fix all these problems. It will be coming soon.

You seriously have no fucking clue.

And yes, the cycle WILL be different

No clue at all. You're a hundred years late to the party.

u/wren42 [+1] (a day later)

Would you be willing to work in a factory in China?

u/ben_oni [+1] (a day later)

I doubt I'd fit in. I don't even speak the language.

All else being equal, would I be willing? Yes. Would I prefer it? prefer it over what?

u/None [+2] (11 hours later)

And unlike humans, EMs wouldn't need all the external stuff we need to keep our bodies running like food and water.

Oh boy, the stuff that I have to unpack from this.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (15 hours later)

Well, they'll need external stuff, but the logistics will be significantly simpler. EMs don't care if energy comes from solar, oil, nuclear, zero point, whatever. People have to get their energy from specific arrangements of hydrocarbons that can't also be too close to other arrangements of hydrocarbons we find poisonous.

u/ben_oni [+1] (18 hours later)

AC, DC, it's all the same, right? Just give 'em whichever.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (18 hours later)

Pretty much, you just need a converter.

u/ben_oni [+2] (17 hours later)

... right! Because computers don't need anything to run. They can be made arbitrarily small, run arbitrarily fast, give off no waste heat, and don't require electricity. Let's see... use the waste heat to power a small steam generator, and use the electricity from that to power the computer! Genius!

In case I wasn't clear enough, I'm mocking you.

In terms of memory storage, human bodies may not be particularly efficient. But that's fine, because they are not primarily static storage mediums, but dynamic systems. Emulating complex systems is usually harder than just letting them run. There are some exceptions: systems that are difficult to observe, very large (or small) systems, or systems that are prohibitively expensive (or dangerous) to run. Humans are basically the opposite of all those.

Scifi writers have been talking about emulated minds for generations. We're a little closer than we once were, but not significantly. In recent years we've seen emulation of a simple multi-cellular lifeform, but that's about it (if I remember correctly, it was something like a roundworm, which has approximately 300 neurons). Wikipedia tells me the human brain has 86 billion neurons, and 10^14 synapses. If each synapse contains a little over one byte of data, that's going to be on the order of 1 petabyte of storage space needed just to emulate a brain. And of course, brains are not particularly fast. Axons don't propagate electrical impulses at the rate of, say, copper, so computations are on the high-µs to low-ms range. But the emulation has to do all this in parallel. Computers are nowhere close to being able to do this. It is, in fact, beyond the theoretical limits of CPU design. You would have to design a custom ASIC capable of running all the calculations in parallel... something like a brain.

Perhaps if everyone were willing to live in extremely slow-motion, emulation could be worthwhile. But it's still not going to be cheaper.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (18 hours later)

... right! Because computers don't need anything to run. They can be made arbitrarily small, run arbitrarily fast, give off no waste heat, and don't require electricity. Let's see... use the waste heat to power a small steam generator, and use the electricity from that to power the computer! Genius!

They don't need to be arbitrarily small or fast or efficient. They just need to be more small or fast or efficient. And in the long term, that's going to be trivial. Sure, moore's law isn't going to keep up its pace forever, but we're far from the absolute limits of physics, and the better AI gets, the faster we can develop chips, which will counteract the fact the low hanging fruit has gotten picked.

Humans are basically the opposite of all those.

Hardly. Humans are plenty expensive over a lifetime in terms of both energy and material costs. Just running a brain in a vat would be more efficient than running an entire human, and at the very minimum there are individual parts of the brain that can be virtualized. We sort of already do this, in fact, with computers standing in for functions like arithmetic and map reading.

Scifi writers have been talking about emulated minds for generations. We're a little closer than we once were, but not significantly.

This is just false. We are exponentially closer, by the nature of Moore's law. Emulating a roundworm is difficult because of the medical and technical aspects of virtualizing a roundworm, and virtualizing humans will be difficult for the same reasons, but in terms of pure computational power we've made utterly massive strides since even the start of the decade. And the thing is, we're already actively working towards solving those issues-- in 2013, Obama was already talking about mapping the entirety of the human brain. We've already have some low-fidelity, low-speed simulations of portions of the brain.

The technology isn't here yet, and the technology won't be here in the next decade, but by the 2030's it'll start to be looking cautiously possible, and I remember doing a calculation based on the processing power of the i7-4790k and moore's law predicting that the absolute lower bound of computing power equivalent to the human brain being affordable for your average joe would happen, by the latest, 2070. Sure, my calculation assumed moore's law would keep on its pace, but I deliberately choose a slower chip than necessary-- the i7-4790k was a few years old as of the time I did my calculation, and compared to contemporary GPUs (which, with their massively parallel architecture are much better for simulating brains than single-digit threadcount CPUs) really quite slow for the purpose of simulating minds.

It is, in fact, beyond the theoretical limits of CPU design.

So the fact that it's beyond the limits of CPU design is a nonstarter. GPUs are what need to be looked at. Plus, GPUs escape one of the big design constraints on CPUs (that it's been more and more difficult to raise single-core performance by increasing clock speed due to heat issues) by just using a shit-ton of CPUs.

Computers are nowhere close to being able to do this.

tl;dr "Nowhere close" doesn't mean much when compared to exponential growth.

u/ben_oni [+1] (18 hours later)

the better AI gets, the faster we can develop chips, which will counteract the fact the low hanging fruit has gotten picked.

This is the epitome of magical thinking.

We are exponentially closer

"Exponentially" is not a unit of measurement. How much closer? Which technical hurdles have been crossed? Saying "computers are faster today" doesn't solve any of the problems. All the models saying it can't be done already account for faster computers in the future. If you have some articles about the low-fidelity, low-speed brain simulations, I would be curious to see them.

my calculation assumed moore's law would keep on its pace, but I deliberately choose a slower chip than necessary

These two things do not offset one another. The base of the exponent and the scalar in front of it are not equivalent parts of the calculation. Given an equation: ke^rt, the k and r terms have completely different impacts on the behavior of the function.

GPUs are what need to be looked at

Do you actually know how GPUs work? Yes, they perform computations in parallel. Do you know how the bus works? How the different computational cores access VRAM in order to operate on input data? GPUs work best when computations can be linearized. Very much like performing graphical operations. And very unlike when emulating non-linear systems like a brain.

just using a shit-ton of CPUs

Nice. How many? How much heat will this kick off? How much cooling will be needed? How much space will it take up? How much latency can we expect between different cores? Again, this is more magical thinking.

"Nowhere close" doesn't mean much when compared to exponential growth

"Nowhere close" means the technical hurdles haven't been crossed. It means that the theoretical limits, as we currently understand them, prohibit this. It doesn't mean I've forgotten about Moore's law. It means that Moore's "law" isn't magic.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

This is going to seem like a cop-out, but to boil it down, the reason I'm so confident that we can get computers of the same processing power and density of the humans brain is because we already have them. The human brain is a 1 to 1 emulation of the human brain. The sounds tautological (and it is) but I very much doubt meat brains are the most efficient possible computing substrate. They're good sure, and they do what we need them to do, but as a fundamental design constraint they're forced to be built using biological processes. A computer has far more options (although not all of them are necessarily cost efficient).

Maybe the brain is somehow a near-ideal substrate, but it's nearly inconceivable that it's impossible to improve on it. Maybe the factor of improvement we can get on it will be low, but as long as there is potential for improvement, and our biological science keeps advancing, there's no reason to suspect we wouldn't have plentiful EMs.

Quite frankly, it seems like wishful thinking to hope that EMs won't dominate the world given enough time.

These two things do not offset one another. The base of the exponent and the scalar in front of it are not equivalent parts of the calculation. Given an equation: ke^rt, the k and r terms have completely different impacts on the behavior of the function.

As an addendum, the i7-4790k was considered the best "bang for your buck" chip at the time. When it got released it was on the latest architecture and printing process (although it was quad core instead of 8/16 core and it didn't have the highest clock speeds), but I did my calculation ~2 years after it got made. That made it a solid basis for comparison. It would be a pain to find my old calculation, but I'd be happy to do it if you gave me your best estimate for the total processing power of the human mind in terms of flops (I saw that it was about 10^16 flops, but I could be wrong) and your timeframe for when an AI-induced technological revolution that would result in the majority of the planet dead would happen.

Obviously the architecture of a CPU and the architecture of the brain are different, but judging by how slow neurons are to transmit signals, it's not inconceivable that some multi-cpu arrangement could simulate humans in at least near-real-time.

edit: taking my figures (I'll wait for yours) since the 4790k runs at 43.98 GFLOPS, and since the doubling in processing power is a discrete rather than continuous process, 10^16 = 2^(x) * 43.98, so x~=21. Which means that it'll take twenty-one doubling periods to get to the rough processing power of a human mind, which means you can set the doubling period to be up to 4 years and still get desktop CPUs matching human brain power by the end of the century. Since the doubling period has been, so far, roughly 2 years, and since server farms would be more than happy to emulate humans (for a price) I can pretty confidently predict that EMs will start popping up well before the end of the century.

u/ben_oni [+3] (a day later)

The human brain is a 1 to 1 emulation of the human brain. The sounds tautological (and it is) but I very much doubt meat brains are the more efficient possible computing substrate.

That's fair, but it's not the whole story.

Maybe the brain is somehow a near-ideal substrate,

It is an ideal substrate. It is the only system capable of perfectly emulating a brain at full speed.

Maybe that's a cop-out, too, but you need the other half in order to keep this sort of argument in balance. And to be fair, emulating a brain doesn't mean emulating all the chemical processes in detail. They can probably be simplified with no noticeable impact in performance.

I'm not arguing that a person can't be emulated in hardware. I'm not even arguing that it can't be done at full-speed. But to think it can be done at full speed on general purpose computing hardware? That's ludicrous. You'll need specialized hardware, specially prepared for the intense multi-tasking operations that are called for. Brains of metal and stone.

Some back-of-the-envelope calculations: 2^47 synapses, with an update period of 1ms. Call it 2^50 updates per second. Each update requires multiple calculations. Let's call it 2^52 flops. This would require around a hundred thousand modern CPUs just to perform the calculations. And we don't have the means with today's technology to handle the shared cache between those processors.

Oh, and my timeframe for an AI-induced technological revolution? I don't think it will happen. I doubt very much that AGI is more than a failed dream.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

It is an ideal substrate. It is the only system capable of perfectly emulating a brain at full speed.

That's not even true right now. The Human Brain < The Human Brain on Coffee < The Human Brain on Amphetamines. Evolution is fundamentally a tinkerer-- it takes a thing we can already do, it tries changing it in a bunch of different ways, and then some of the things stick and get passed on to the next generation. But while evolution is great at finding local maximums, it's fundamentally incapable of finding global maxima.

Well, perhaps I should take a step back. Yes, human brains are and likely will be the best device for emulating an absolutely 100% accurate human brain, but there's a difference between a "brain" and a "person". There's a bunch of stuff in my brain that I need, but isn't exactly me. A slightly different autonomic response system wouldn't change who I am. So if you remove that "100%" constraint, even in favor of a 99.99% constraint instead, there's a fair bit of leeway with regards to implementation details.

You'll need specialized hardware, specially prepared for the intense multi-tasking operations that are called for. Brains of metal and stone.

I don't necessarily disagree, but that doesn't actually preclude the existence of massive amounts of EMs. The demand for them would be simply massive, if only for the simple reason that the kinds of people comfortable with virtualizing themselves are the kind of people comfortable with spinning up as many variations of themselves as allowed by budget and computing power.

Some back-of-the-envelope calculations: 2^47 synapses, with an update period of 1ms. Call it 250 updates per second. Each update requires multiple calculations. Let's call it 2^52 flops.

That works out to ~4.5*10^15 flops, which is actually more permissive than my estimate. And remember, my estimate was for desktop chips.

When it comes to clusters of computers, well, as of 2012, google had ~40*10^15 petaflops of processing power available to it, and of course that number has only grown (exponentially) since then. Yeah, that was the computing power of a megacorporation, but currently, the US and China are competing to build an exascale (10^18 flops) supercomputer. And again, computers have been getting better, cheaper, and more efficient for more than forty years straight, and the process shows no signs of stopping (even if it does show signs of slowing down.)

Sure, we don't have the means with today's technology to handle the shared cache, but that's an engineering issue I'm more than confident will eventually be fixed. After all, there's no point to having technology to handle that massive cache when we don't have the tech for it anyways.

Oh, and my timeframe for an AI-induced technological revolution? I don't think it will happen. I doubt very much that AGI is more than a failed dream.

Then do you believe the majority of people will die in a techno-revolution without the use of AI? Because the whole premise of the OP was that the only way the majority of people would survive the techno revolution would be through subsistence farming, and my counter was that any techno-revolution that would kill off the majority of meatspace would have so many EMs around that the majority of people period would still be alive.

Also, AGIs exist now. We call them "humans". Facetiousness aside, we know that AGI is possible and because of evolution, we know that it's possible for lesser minds to develop more intelligent minds. Sure, the natural processes for AGIs to arise take a few billion years, give or take, but we already have a good starting point.

u/ben_oni [+1] (a day later)

Also, AGIs exist now. We call them "humans".

...What? Did you lose brain cells during this conversation? I think you might be broken.

There's a bunch of stuff in my brain that I need, but isn't exactly me.

No. That's all you. Everything is intrinsically connected. Start changing the body, and the brain changes, too. Change something in the brain, and the body changes. You can't have it halfway.

A slightly different autonomic response system might not change who you think you are, but it will change what you are. Pull out enough "useless" stuff, and eventually you'll have a machine that thinks it's you while actually being something else entirely. Probably a p-zombie.

and the process shows no signs of stopping (even if it does show signs of slowing down.)

What do you think slowing down is? You see a car slowing as it approaches a stop sign. You think to yourself: "That car has been going pretty fast for quite some time; I don't think it's really gonna stop; it'll probably just roll through."

Improvements can reach an upper limit without ever "stopping". f(x) = x/(x+1) never stops increasing. Not hard to see that it will never reach 1.

supercomputers

One word: Latency. Did you know that throwing more processors at a real-time problem doesn't actually make it easier?

shared cache

that's an engineering issue I'm more than confident will eventually be fixed

Here's an actual engineer telling you it is insurmountable. I've explained how to handle it: Specialized systems.

we know that AGI is possible and because of evolution

No, we don't. We know that meat-form intelligence works. We don't know anything else that can work. Maybe we can simulate a human brain. If we can (and that's a big if), it would be strong evidence that AGI is possible.

Then do you believe the majority of people will die in a techno-revolution without the use of AI?

No, I don't. And I don't really care about what OP wrote. That guy's probably a moron anyways. What I am discussing is the concept of emulated minds and that even though they are a perfectly nice fictional concept, they are not one that works out in the real world. We can revisit the topic in twenty or thirty years, and see if any advancements have proven one of us right or wrong.

u/crivtox [+2] Closed Time Loop Enthusiast (a day later)

No, we don't. We know that meat-form intelligence works. We don't know anything else that can work.

Yes but you need some reason for asuming that meat form intelligence is the only thing that can work, you can't just assume anything we don't know how to do yet is impossible until otherwise proven , and its a really weird claim that the way evolution found is either the only way to do it or the best way to do it ( This almost never happens because evolution is like a hill climbing algorithm ) .Can you explain your specific reasons why do you think this is the case with intelligence?.

u/ben_oni [+1] (a day later)

These are good points, and I have to admit I go back and forth on the issue, depending on how cynical I am at the time. I'll just give one argument right now that is likely to appeal to the r/rational community: the anthropic principle.

If synthetic intelligence is possible, we would expect, as u/GaBeRockKing has been arguing, that they would be the dominant form of life. Since I find that I am a meat-form intelligence and not a synthetic intelligence, I must consider that either synthetic minds are rare throughout all of time and space (relative to expectations); or else that most synthetic minds begin existence as a meat-form intelligence.

Alternatively, perhaps I am a synthetic intelligence that thinks it is a meat-form intelligence. This would indicate that the universe as I know it is probably a simulation of some sort: in that case, I don't know the rules, can't know the rules, and speculation is pointless.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

If synthetic intelligence is possible, we would expect, as u/GaBeRockKing has been arguing, that they would be the dominant form of life. Since I find that I am a meat-form intelligence and not a synthetic intelligence, I must consider that either synthetic minds are rare throughout all of time and space (relative to expectations); or else that most synthetic minds begin existence as a meat-form intelligence

I explained this before: evolution is great for finding local maxima, not so much for finding absolute maxima. Take the human larynx, for example. Sure, it's passable, but compared to the range of sound a cheap modern speaker can make, is completely inadequate. But there's no evolutionary mechanism for humans to suddenly develop electronic larynxes-- evolution is stuck tweaking hundreds of millions of years old tech.

And it's not uncommon to find evolutionary dead ends, where a species hyperoptimizes for some local maximum, then goes extinct because an invasive species optimized for some better local maximum comes in and messes them up.

It's a misapplication of the anthropic principle to ask why we haven't been replaces by nonbiological brains yet, because the answer is that only recently have the conditions become favorable for the emergence of mechanical minds.

It's like asking "well if humans are so great, then why weren't they around for the first billion years of evolution on earth? Obviously, humans can't exist becase they haven't existed yet."

u/ben_oni [+1] (a day later)

You misunderstand. The anthropic principle applies, but it only offers weak evidence, not strong. Obviously, there must be a precursor. But what are the odds that you personally would be in the precursor group?

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (2 days later)

But what are the odds that you personally would be in the precursor group?

To be fair, from a pure anthropic perspective, you'd be right. But from a frequentist perspective, literally everything not alive right now has been part of a precursor group, and I have no reason to suspect I'd be deviating from that trend.

u/GaBeRockKing [+1] Horizon Breach: http://archiveofourown.org/works/6785857 (a day later)

...What? Did you lose brain cells during this conversation? I think you might be broken.

So humans aren't human-created general intelligences? Color me surprised.

No. That's all you. Everything is intrinsically connected. Start changing the body, and the brain changes, too. Change something in the brain, and the body changes. You can't have it halfway.

A slightly different autonomic response system might not change who you think you are, but it will change what you are. Pull out enough "useless" stuff, and eventually you'll have a machine that thinks it's you while actually being something else entirely. Probably a p-zombie.

P-zombies don't exist. If something thinks its sapient, then it is. Now, that's not to say that sufficiently invasive procedures would be enough to effectively count the prior me as "killed" and the modified person as a new person, but the threshold for that has to be considerably higher than your envisioning, or puberty would kill the original person and replace them with a new one. The same goes for stuff like nootropic drugs, brain damage, and even just the effects of aging.

What do you think slowing down is? You see a car slowing as it approaches a stop sign. You think to yourself: "That car has been going pretty fast for quite some time; I don't think it's really gonna stop; it'll probably just roll through."

I know as well as anyone that all exponential curves will eventually turn logistic, but there's just no evidence that the computing power curve will stop being exponential before technology surpasses the human mind. Again, it's wishful thinking to believe the human brain is as good as it gets.

One word: Latency. Did you know that throwing more processors at a real-time problem doesn't actually make it easier?

So what if they have latency? The nervous system triggers along snail-slow chemical paths. Fiber optic busses will be, quite literally, a lightspeed improvement on what our biology is capable of. Latency is hardly insurmountable.

Here's an actual engineer telling you it is insurmountable. I've explained how to handle it: Specialized systems.

"This thing is insurmountable, which means we need to surmount it with specialized systems." ?????

You're contradicting yourself.

No, we don't. We know that meat-form intelligence works. We don't know anything else that can work. Maybe we can simulate a human brain. If we can (and that's a big if), it would be strong evidence that AGI is possible.

We will be able to simulate a human brain. That's literally what I've been arguing this entire time. Maybe not at 100% accuracy, but we don't need 100% accuracy for the simulation to still be a person. The human brain is special because it's better than everything that's come before, not because of some inherent quality that's impossible to replicate.

No, I don't. And I don't really care about what OP wrote. That guy's probably a moron anyways. What I am discussing is the concept of emulated minds and that even though they are a perfectly nice fictional concept, they are not one that works out in the real world. We can revisit the topic in twenty or thirty years, and see if any advancements have proven one of us right or wrong.

Ok then. Just remember, though: everyone who's previously doubted the ability of computers to keep getting faster has been proven wrong. That doesn't necessarily mean they won't be, but making decisions that bet on the human mind remaining the king of the hill in any particular domain have tended to be proven wrong so far.

u/FeepingCreature [+1] GCV Literally The Entire Culture (10 hours later)

Yeah but Ems will be specialists, cloned over and over. It's not a viable path for the overwhelming majority of the populace because supply vastly outstrips demand.

u/Roxolan [+3] Head of antimemetiWalmart senior assistant manager (7 hours later)

Oh sweet. I've thoroughly enjoyed (and learned from) his various "civilization inadequacy" facebook posts, very happy to see a longer piece on the topic.

u/ElizabethRobinThales [+3] Practically Perfect in Every Way (a day later)

Is it weird that the first thing I thought of when I saw the cover was the poster for the Mary Poppins musical?

---