16 KiB
[D] How robust are human terminal goals on the ascent toward trans-humanity ?
-
Author: u/VanPeer The shard made me do it*
-
URL: https://www.reddit.com/r/rational/comments/4b0b1l/d_how_robust_are_human_terminal_goals_on_the/
-
Score: 9
-
Created: 2016-03-18T21:23:33
Post:
The title question is something I have wanted to explore for a while in fiction. [TAP] (http://www.infinityplus.co.uk/stories/tap.htm) by Greg Egan feels like an excellent reference point, though the title of the post only tangentially relates to the story.
disclaimer: I'm a huge fan of Egan and this post is as much a chance to link to his older work.
Comments:
u/SvalbardCaretaker [+17] Mouse Army (32 minutes later)
The prevailing fear is that they might not at all be robust. Value drift seems to be a real danger.
Also the question of wether different parts of humanity actually do have convergent or divergent terminal goals is as of yet hotly debated.
Eliezer argues pro convergence here: http://lesswrong.com/lw/rl/the_psychological_unity_of_humankind/
Some anthropologist on LW once strongly argued otherwise, and a few years ago Lukeprog said he wasnt sure, but leant towards divergence.
Other good starting points are, eg:
http://lesswrong.com/lw/y3/value_is_fragile/
http://lesswrong.com/lw/y4/three_worlds_collide_08/ (Fiction)
u/VanPeer [+2] The shard made me do it (4 hours later)
Thanks for the response. I'll chew on it for a while.
u/roystgnr [+1] (3 days later)
There is a third possibility: divergent-but-negotiable. The present world is one in which I often give up some of my goals in favor of others' and vice-versa, but in each trade off we're both still happy, and it's possible that this will remain a stable equilibrium.
u/ArgentStonecutter [+3] Emergency Mustelid Hologram (an hour later)
Since no human actually knows what their own goals really are, since they have no direct visibility into their own decision-making process, this is kind of unanswerable.
Since you're a Greg Egan fan, might I direct you to Mister Volition?
u/VanPeer [+1] The shard made me do it (4 hours later)
Thanks! Looks interesting. Here is another good one, in case you haven't read it: [Singleton] (http://gregegan.customer.netspace.net.au/MISC/SINGLETON/Singleton.html)
u/None [+3] (26 minutes later)
What's your actual question? Humans don't have built-in terminal goals.
u/VanPeer [+6] The shard made me do it (3 hours later)
Sorry for being unclear. I was wondering how plausible a certain fictional scenario is: a male protagonist has what he considers his core values acquired through nature or nuture. To himself, his values feel right. Then, he gets augmented with the ability to scrutinize his feelings more deeply (read only access), and starts perceiving his "core" feelings as a composite of more basic drives. His original values no longer seem so... sublime. Reasonable ? Laughable ? I'm not sure. For example, understanding the evolutionary roots of love doesn't make one stop loving his family. On the other hand, there may be no evolutionary defenses against trans-human self insight.
u/None [+3] (4 hours later)
What's wrong with self-insight?
u/VanPeer [+5] The shard made me do it (4 hours later)
Nothing per se. I was just wondering if augmented self-insight might plausibly cause divergence from core values that exist before the augmentation. For example, if a male protagonist gets superhuman insight into his deeply cherished feelings of gallantry towards females, and then realizes his gallantry to be nothing more than sublimated physical attraction, might he not simply lose interest in being 'gallant' ?
u/None [+5] (6 hours later)
That sounds like exactly the situation in which he should stop being gallant.
u/None [+5] (8 hours later)
[deleted]
u/TK17Studios [+5] Author of r!Animorphs: The Reckoning (9 hours later)
Not sure if the use of the word "good" was irony, though the TM implies you're aware of it.
This is the Dr. Manhattan problem, right? The question becomes relevant if some humans are ascending and others aren't. If the only individuals "left" are ones following this path, though, then it seems more severe than, but ultimately similar to, our species' moral development thus far. We largely toned down our enslaving, raping, and murdering of one another; if we discover that other things we thought were true aren't, we'll change again/more.
I would wager some terminal goals are more robust than others. If you gave cavemen heroin, they'd probably go nuts for it and see nothing wrong with it; our distaste for wireheading comes with knowledge and perspective, and allows us to look at it and say, kinda pathetic, huh? Maybe we can get happiness that's more justified and legitimate and "real" some other way. Similarly, a lot of our current sources of happiness may become more transparently trivial or mechanistic, and lose some of their charm. But I would expect values like curiosity to retain their allure, and values like kindness to shift in justification but remain relatively strong/popular (swapping good feels for decision theory, for example).
u/chaosmosis [+5] and with strange aeons, even death may die (9 hours later)
I think the majority of our disgust for wireheading might be self-delusion and social fiction. I sort of think there is nothing but wireheading and nihilism, and different points between the two extremes.
u/TK17Studios [+3] Author of r!Animorphs: The Reckoning (10 hours later)
Maybe. I respect wireheading as a force that could take decisive control away from "me" ... I don't kid myself that I could just not be addicted to heroin, for example. But from this perspective, I have a what-seems-to-be principled stand against deriving satisfaction from the intrinsically meaningless.
u/chaosmosis [+3] and with strange aeons, even death may die (10 hours later)
Do you also have a principled stance against eating chocolate? Chocolate seems rather intrinsically meaningless to me.
u/TK17Studios [+2] Author of r!Animorphs: The Reckoning (18 hours later)
I do have a gradient, which is perhaps unjustified, but I lean much further away from random hedonic pleasures (sex, alcohol, drugs, video games) than average, for basically this reason.
u/None [+1] (17 hours later)
Define: "meaningless".
u/chaosmosis [+2] and with strange aeons, even death may die (18 hours later)
The word "meaningless" here refers to whatever is meant when people say that wireheading is bad.
u/lehyde [+2] Nudist Beach (11 hours later)
So why can't we desire a specific point between wireheading and nihilism? There is no reason our desires should coincide with one extreme, right? Maybe there is a sweet spot.
u/chaosmosis [+2] and with strange aeons, even death may die (18 hours later)
I didn't mean the last part of what I said. I should have said only "there is just wireheading and nihilism". Any points that appear to exist between the two positions are due to human cognitive failings. The only reason we haven't all wireheaded ourselves already is technological limitations. Even with our current tech level, most people's free time is spent in escapism through fiction, drinking alcohol, or getting endorphin rushes from exercise, competition, and sex.
I even think the inevitability of wireheading might be a good explanation for Fermi's Paradox.
I would agree that it is desirable that we maintain our social fictions surrounding wireheading and nihilism, though. But that's why efforts to improve human rationality to transhuman extents make me so nervous. I think they will undermine a lot of the meaning in people's worldviews, including my own. At the same time, doing nothing and refusing to push our limits also seems intolerable to me. So it's a bit of an unwinnable situation.
u/VanPeer [+1] The shard made me do it (a day later)
I concur. But that shouldn't stop us from self improvement. If the human race collapses into a wire headed singularity, so be it.
u/Bowbreaker [+1] Solitary Locust (a day later)
I would agree that it is desirable that we maintain our social fictions surrounding wireheading and nihilism, though.
And why is this desirable for you? Not that it shouldn't be but this terminal value of not actually devolving to one of the binary states of wireheading and nihilism must come from somewhere, no? Or do you think that this is only your own self-protecting delusion and the only reason you are not in favor of wireheading/nihilism is that you have yet to achieve overcoming your own very powerful biases surrounding it?
u/VanPeer [+2] The shard made me do it (16 hours later)
I suspect the same. Recently read an excellent piece on wire heading by Scott Alexander, but it was never clear why exactly wire heading was bad as long as one doesn't have dependents who would be abandoned, and if one is fully aware of the consequence of pressing the button.
u/pleasedothenerdful [+1] (3 days later)
Link?
u/VanPeer [+1] The shard made me do it (11 days later)
[No Time Like the Present] (http://slatestarcodex.com/2015/05/29/no-time-like-the-present-for-ai-safety-work/)
u/pleasedothenerdful [+1] (13 days later)
Wireheading is certainly a failure mode in that it becomes the only remaining terminal goal of the participant, and so prevents the accomplishment of anything that isn't wireheading. The wired stimulation of pleasure centers is so much stronger that the normal pleasurable feedback of biologically- and socially-necessary processes like eating, drinking, and hygiene are no longer motivational. There is nothing left but The Button, and, shortly after, a dehydrated, urine-soaked corpse.
Scott's article was discussing it not just as a possible failure mode of all reinforcement-learning-, reward-function-based minds but as one that a self-modifying, superintelligent AI might be particularly prone to. If that AI's reward function is such that it could be maximized by conversion of the local biosphere into computronium or paperclips or grey goo or self-replicating probes--and that doesn't seem terribly unlikely--then that seems like it would be pretty bad, even though it's really "just" wireheading.
Kind of like hardcore drug addiction, in a post-scarcity society humans wireheading might not actually be the worst thing ever (although still extremely selfish and self-indulgent), but, short of that, it's a real drain on society.
Either way, to anybody with goals other than wired-self-stimulation of one's own pleasure centers, wireheading is suboptimal. People tend to will toward generalization of their own beliefs ("everybody should believe and think and act the way I do"), so anybody who isn't doing it already is probably going to oppose it as a valid terminal goal for others by default. Everybody who is doing it already isn't going to make any noise one way or the other except clickclickclickclickclickclickclickclickclickclickclickclick.
Anybody with any sort of teleological beliefs or ethics is also going to look down on it. That teleology doesn't even have to be religious; it's not hard to think of potential purposes for a human life that are more noble—or even just less all-encompassingly self-centered—than spending it doing nothing but stimulating one's own pleasure center with an electrode. Even if the actual, biological reason anybody does any of those other things is really just to stimulate that pleasure center, albeit less efficiently.
Heck, I think even a consequentialist doing the math on "greatest pleasure for the most people" is going to come up with something other than "everybody wirehead, starting right now" as the optimal answer, even if that same consequentialist might not be so opposed to digitization of all existing sentient minds, virtual wireheading, and conversion of the solar system into computronium to maximize the number of duplicate minds that can virtually wirehead simultaneously.
Until we hit post-scarcity and have benevolent AI gods to make sure we don't die while we're all wireheading, everybody going on as normal and not wireheading will likely result in greater net pleasure than everybody wireheading for a few days or weeks before the extinction of the human race.
u/VanPeer [+3] The shard made me do it (16 hours later)
Well said. Gives one more hope for the future.
u/None [+3] (17 hours later)
This is the Dr. Manhattan problem, right?
Except that Dr. Manhattan was psychologically and scientifically unrealistic. The differences between a live human being, a dead human being, and a merely brain-dead human being are very palpable and scientifically detectable. There does not exist a level of Deep Truth at which these differences disappear, only levels of Deep Truth to which the merely ordinary, everyday truths are reducible.
If you gave cavemen heroin, they'd probably go nuts for it and see nothing wrong with it
Would they? Even rats don't seem to actually prefer opiates or wireheading if they're offered a desirable alternative.
u/None [+2] (17 hours later)
It may be the case that with enough self-insight, you realize that much of what we call morality is merely an extension of the desire to conform.
Yes, that is why people consider it "decent and moral" to wear pants in public. It is a conformity thing.
For example, Emily's core values include the protection of conscious life, but upon augmented introspection, she realizes that a collection of atoms acting one way is not fundamentally different from or more meaningful than atoms acting in another way.
That doesn't sound like augmented introspection. That sounds like artificially-installed nihilism.
Is this plausible? Conceivable?
Well, as implied above, I find it rather implausible.
Is self-insight always, necessarily, a good thingTM?
Yes, more knowledge is always helpful for making decisions that are better in-tune with reality.
u/VanPeer [+1] The shard made me do it (16 hours later)
Social conformity is a good point. Hadn't thought of that.
u/callmebrotherg [+1] now posting as /u/callmesalticidae (9 hours later)
Seconded. Not sure what the problem is here.
u/ArgentStonecutter [+1] Emergency Mustelid Hologram (13 hours later)
That's kind of the point of Mister Volition.
u/Izeinwinter [+2] (21 hours later)
Heck, it's entirely possible that terminal values are robust, but that enhancement would ruin your ability to be inconsistent, leading to hilarities like upgraded humans doing the math on suffering and promptly converting the entire biosphere into computronium as an animal welfare measure, because they loose the ability to be okay with feeding cows to dogs, mice to cats, and krill to whales.
u/Eryemil [+1] (a day later)
You're OK with that?
A long time ago I realised that for values to be internally consistent only abolitionism made sense. How it is achieved is not the point. Digitisation is not even the most radical solution I've considered.