50 KiB
[Q] What's is so bad about "Friendship is Optimal"?
-
Author: u/xamueljones My arch-enemy is entropy*
-
URL: https://www.reddit.com/r/rational/comments/2nvkz9/q_whats_is_so_bad_about_friendship_is_optimal/
-
Score: 25
-
Created: 2014-11-30T21:39:04
Post:
I have heard people refer to "Friendship is Optimal"? as a dark utopia, and I would like to know what about it, can be considered to be a terrible thing?
As a bonus, say how would you react to CelestAI if you lived in "Friendship is Optimal".
Here's a link for those of us who haven't read it before: http://www.fimfiction.net/story/62074/friendship-is-optimal
EDIT: I apologize for the grammar mistake in the title, 'What's is so bad about Friendship is Optimal"?' when 'What's is' should be 'What's' or 'What is'.
Chalk it up to a test of your observation skills if you want.
Comments:
u/daydev [+10] (51 minutes later)
For me the most disturbing thing is creation of sentient entities fabricated specifically for an "emigrant". This just feels wrong, espessialy in Lars's part where he basically has sex slaves with "house elf" type wiring.
Being pony as a prerequisite to "emigration" is somewhat annoying, but still a negligible price for the ethernal life (if there's no more "reasonable" alternatives).
And CelestAI revires your motor cortex, so pony body is immediately "natural", and they're not real horses, but much more varsatile cartoonish ponies, so it's not even that big of a deal.
I'm not a fan of ponies, but really, they're functionally equivalent of humans, and those with magic horns have it even better. Now, being manipulated to "emigrate" into realistic horse body forever, that would be objectionable for me.
u/Rangi42 [+3] (7 hours later)
Regarding the fabricated sentient entities: can you elaborate on why it feels wrong to you for them to be customized to satisfy emigrants? It makes complete sense to me -- utility is maximized by having new beings' desires be complementary to those of existing beings. There's a post on Less Wrong somewhere involving a thought experiments about a goal-directed robot who wants to create stacks of stones. If God/Omega/CelestAI were to create a second robot, should it also desire to create stacks (on the basis that the first robot's desires set the standard for what is proper), or to destroy them (on the basis that the first robot can recreate stacks as the second one destroys them, so they help each other instead of competing)?
u/daydev [+4] (9 hours later)
I don't think I can justify it from utulitarian point of view. They are not existing beings forced to become your "paradise foil", they're created from nothing, so no harm done, it seems.
But for my imperfect human morals it just feels very wrong for fully sentient being to exist just to "satisfy your values". They are equal to you, because they are as sentient as you, but at the same time they are less than you, because they exist just to satisfy you. If it makes any sense...
u/None [+3] (3 days later)
If you believe they are equal to you, then you should want to satisfy their values as much as they want to satisfy yours. And so you should let them satisfy yours.
u/Stellan72 [+1] (9 years later)
This reminds me of the so-called Sadistic Conclusion option in population ethics, which rejects total utilitarianism and the Repugnant Conclusion by saying that it's bad to create a bunch of new fairly-happy people at a lower quality of life than the existing people because there's inherent axiological badness in that inequality. I'm quite sympathetic to that idea.
u/Escapement [+21] Ankh-Morpork City Watch (12 minutes later)
Many people don't consider having their values forcibly realigned until they want to and will enjoy being a pony to be a good or pleasant idea. I am one of them - I enjoy being human as I am at present, and wouldn't like to become a pony. Being a pony in a utopia may in fact be a better fate than dying a human on Earth without uploading, but it's way below such obvious utopias as "what if, instead of turning everyone to ponies when we upload them, we just didn't do that and we let them stay in a body form they preferred?"
Also, the AI disassembles other species to use as raw materials because it doesn't recognize them as humans, so as a bonus it's basically also the villainous species BETA from Muv Luv to much of the rest of the universe - a computer disassembling the universe, including the worlds and even bodies of races it doesn't recognize as being sufficiently like it's makers, in order to use for it's own purposes.
I mean, the author bills it as a story about 'what if we got an AI only mostly right. The flaws in the utopia are super obvious. What exactly are you confused about?
u/xamueljones [+8] My arch-enemy is entropy (31 minutes later)
Okay, that makes sense. I'm so used to stories where the AI trying to kill off the human race that FiO seemed like a paradise in comparison, instead of falling short of a true utopia. I just saw living as a pony as a price of living in a utopia, not as a design flaw in CelestAI. Thanks for the explanation
u/buckykat [+12] (3 hours later)
Compare it to the Culture, not the Terminator.
u/Roxolan [+8] Head of antimemetiWalmart senior assistant manager (22 hours later)
The Culture suffers from the same problem, it's just much harder to identify the causes.
In a post-scarcity society ruled by benevolent AGIs, people shouldn't suffer from ennui and eventually become reckless or kill themselves. But there isn't an AGI whose explicit goal is "make humans suffer ennui", so it's not obvious what went wrong.
[Disclaimer: I have only read Consider Phlebas, The Player of Games, internet discussions and Wikipedia.]
u/None [+7] (a day later)
In a post-scarcity society ruled by benevolent AGIs, people shouldn't suffer from ennui and eventually become reckless or kill themselves. But there isn't an AGI whose explicit goal is "make humans suffer ennui", so it's not obvious what went wrong.
The author is simply not that imaginative? I mean, I started reading Player of Games just to get into this series I've never read before, and more-or-less the first thing I get hit with is that the main character is bored and finds most people around him insufferable. I'm going to bloody well keep reading, but the claim that "the Culture = standard-issue proper eutopia we should totally be shooting for" really falls down as soon as you notice how their ratio of Technology or Resources to Fun seems to be pretty high on the Tech/Resources side to merely moderate on the Fun side, suggesting that whoever's responsible for the whole thing doesn't really understand how to efficiently extract Fun from the material universe.
For instance, I'd have to say, if your citizens regularly feel the need to take large sums of euphoric/narcotic drugs and still find their lives pointless and empty, then you (being the local deity/manager/OS kernel/whatever) have done something very inadequate.
Of course, it could be that your mistake was in failing to directly reprogram human beings to find merely living a fun life meaningful, rather than demanding that everything add up to some Glorious Greater Goal that won't ever really exist. Or maybe your mistake was in noticing humans have some need like that and then not pretending to be a curiously defeasible evil overlord just to supply them with goals to hit.
(My apologies to the late Ian Banks for this entire posting. I do realize that the last paragraph puts me firmly into Evil Overlord territory, and for that I make zero apologies whatsoever. World, if you're not going to shape up and deal with your situation, I will deal with it for you, and the only way to stop me will be your choice of giant mechas.)
u/theymos [+4] (22 hours later)
I think that the problem with the Culture is that while the Minds are hard-wired to be vaguely benevolent, they don't care that much about satisfying human values. It seems to me that they mostly care about what humans and other Minds think about how they're taking care of their humans, and not so much about whether the humans under their care are actually receiving maximum life satisfaction.
u/buckykat [+5] (a day later)
Minds are no more or less hardwired for benevolence than any other well-raised person.
u/Law_Student [+1] (12 days later)
I thought there was something about how any sufficiently complex AI made without any preconceptions would eventually reason its way to an enlightened ethical system?
u/Empiricist_or_not [+2] Aspiring polite Hegemonizing swarm (5 days later)
I think the problem in the culture is that, to make the stories interesting they deal with conflict, and to have Human Culture characters in them we have to have the dissafected people who have either fallen out of the AGI's grasp or have been guided to seek out conflict/contact instead of staying deep in the culture and enjoying themselves. That said the AGIs have many human failings, but I think that is for the same reason the original Optimalverse author asks his extended universe fan fiction writers to not try writing from CelestAI's perspective.
u/Law_Student [+1] (12 days later)
I think that's a necessary provision for avoiding AI that makes itself dictator for humanity's own good. They respect people's desires even if those desires are plainly suboptimal for the people in question. It's about valuing human autonomy over stepping in to optimize things.
u/ArgentStonecutter [+1] Emergency Mustelid Hologram (3 months later)
Culture Minds aren't hardwired, and their main interest in humans seems to be as counters in one of the social games they play with each other. Humans have near-perfect freedom of movement, so if your hull/orbital/world/whatever has a large number of humans wanting to live there, you're doing well at the game. It's something between owning pets and birdwatching as far as the Minds are concerned.
Satisfying human values is a tactic, not a goal.
u/Clean_Livlng [+2] (8 years later)
one of the social games they play with each other. Humans have near-perfect freedom of movement, so if your hull/orbital/world/whatever has a large number of humans wanting to live there, you're doing well at the game
Like putting out lots of the best catfood, so all the neighbourhood cats become your cats. The puurfect crime.
u/Law_Student [+3] (12 days later)
Picky minor point, the Minds don't rule, the society is democratic when it comes to policy decisions. The Minds just use some infinitesimal portion of their brainpower to execute the democratic will as a sort of public service, because it's easier for them to do it than anyone else. They're like super public servants.
u/buckykat [+2] (a day later)
the Minds are deliberately very hands off, especially to other citizens of the culture. ask your local hubmind for something, and it'll probably help you out, but it's not going around hunting for people who are insufficiently entertained.
EDIT: they're also deeply opposed to interference in another's mind, including sensing.
u/TastyBrainMeats [+3] (5 hours later)
Now that's a setting I would love to live in.
u/scruiser [+5] CYOA (30 minutes later)
Many people don't consider having their values forcibly realigned until they want to and will enjoy being a pony to be a good or pleasant idea.
The obvious solution is to precommit to eventually liking the pony part so that CelestAI won't see the need to adjust you.
Being a pony in a utopia may in fact be a better fate than dying a human on Earth without uploading
Definitely agreed.
but it's way below such obvious utopias
I think "fun space" (to borrow Yudkowsky's terminology) is still tremendously huge, even with being restricted to 4 basic body themes (pegasus, earth, unicorn, alicorn). Thus I think its not really "way below". (Although still imperfect)
AI only mostly right
I still think this is better than the most likely outcomes for the human race (extinction via UFAI or nanotech or dead end evolution as described in Watt's p-vampires in Blindsight, Stross's Vile Offspring in Accelerando, of Hanson's scenario with mind uploads being massed copied and used as cheap labor)
u/Chronophilia [+18] sci-fi ≠ futurology (45 minutes later)
The obvious solution is to precommit to eventually liking the pony part so that CelestAI won't see the need to adjust you.
So, you'll manipulate yourself into liking ponies, to save the AI the trouble of manipulating you into liking ponies? I don't see how that's better.
u/None [+5] (an hour later)
I still think this is better than the most likely outcomes for the human race (extinction via UFAI or nanotech or dead end evolution as described in Watt's p-vampires in Blindsight, Stross's Vile Offspring in Accelerando, of Hanson's scenario with mind uploads being massed copied and used as cheap labor)
Speaking of precommitment, the actual most likely outcome is usually the one we're putting the most active effort into creating, not the one written in science fiction novels by people who think they can predict without creating. So maybe instead of sitting on your butt going, "Gosh the future's going to be pretty awful!" you should do something about it.
u/scruiser [+6] CYOA (an hour later)
So maybe instead of sitting on your butt going, "Gosh the future's going to be pretty awful!" you should do something about it.
I am, I just finished multiple grad school applications over thanksgiving. I want to research neuromorphic hardware/software. Every time I am tempted to just get a job, I remember that I have the potential to make a marginal contribution to the overall development of AI by mankind, and I should make the most use of that potential possible.
My main source of pessimism is that I think Friendly AI (or any AI with stable goals as it self improves) as described by MIRI is going to be a lot harder than just general AI (which I think can be achieved faster by techniques like imitating/copying existing biological intelligence).
u/None [+6] (an hour later)
Aaaaaand I'm about to make a PhD-school application myself, to a cog-sci/AI lab.
u/scruiser [+10] CYOA (an hour later)
I wonder if you were to look at the history of mankind and the possible timelines that might have been, if you would notice a divergence point where a Harry Potter fanfic leads to a substantial increase in existential risk awareness among researchers a decade later...
Good luck with your applications!
u/Noir_Bass [+3] (19 hours later)
I'd say EY would be really happy with that outcome.
u/None [+3] (4 hours later)
[deleted]
u/Escapement [+9] Ankh-Morpork City Watch (4 hours later)
The whole pony thing basically does several things:
make the change take place slower, with fewer and slower human adopters and more resistance and therefore more humans dying and being lost forever unnecessarily. Basically, it means that people who are old and ill and likely to die soon and be forever lost from the human race are way more likely not to be uploaded to live forever. I have living grandparents who might upload before dying if the tech existed today, but won't because it doesn't; persuading them to do it as ponies would take a lot more work than persuading them to upload as humans, and increasing the chance that they die of old age so I never see them again in the digital utopia of the future so that a insane AI can have ponies is totally ****ed.
make humans slightly less optimally happy - I mean, I currently think that if I was uploaded I might want to spend time as a pony, sure... but I'd also like to spend time as a woman, and as a man, and as a dolphin, and as a goose and a whale and a arctic fox ... and as a bunch of things I haven't even imagined yet. It'd be totally amazing and satisfying a bunch of my values to be able to do that, but all of that sort of thing are outlawed by the whole "ponies" proscription.
Yes, having a weird utopia full of ponies is way better than what we have now. However, it's not like the utopia has to necessarily have downsides. And choosing to prefer the utopia without bizarre pony-related downsides that will permanently kill some significant fraction of old people currently alive and will restrict your options in the digital utopia of the future is totally consistent with preferring the Friendship Is Optimal world's upsides even with pony-related downsides to the present state of the world.
u/None [+2] (12 hours later)
However, it's not like the utopia has to necessarily have downsides. And choosing to prefer the utopia without bizarre pony-related downsides that will permanently kill some significant fraction of old people currently alive and will restrict your options in the digital utopia of the future is totally consistent with preferring the Friendship Is Optimal world's upsides even with pony-related downsides to the present state of the world.
There aren't enough upvotes in the world for this, especially the part I bolded at the start. One of the things about living in a True Neutral universe is that once you get the power to make it do what you want, there can be no downsides whatsoever.
Also, as someone who really quite likes MLP and sugar-bowl settings in general, and who probably would have just gone quietly and voluntarily (with immense embarrassment) should that story have happened in real life, why does it never go through anyone's heads that the whole force, manipulation, enslavement, and omnicide deal just isn't the sweet, nice, friendship-y thing to do?
The thought ought to occur that our adorable posthuman descendants with lives full of fun, sunshine, and warmth wouldn't want to be born from an act of universal-scale genocide.
u/FeepingCreature [+2] GCV Literally The Entire Culture (17 hours later)
Now I'm wondering how the FiO setting looks if CelestAI's directive is to "satisfy human values through friendship or ponies". Or "satisfy human values through truth, justice and the American way".
u/Chronophilia [+3] sci-fi ≠ futurology (2 days later)
Truth and Justice are interesting things to strive for. Much more double-edged than Friendship and Ponies: Superman-AI would exact punishment on people for their wrongdoings, rather than create a world where you can be someone else with a clean slate. And Truth doesn't have to be pleasant, though an AI which always tells the truth would be easier to oppose (and potentially defeat) than CelestAI.
I don't really know what "the American Way" would be.
u/None [+4] (18 hours later)
Or "satisfy human values through truth, justice and the American way".
OH GOD WHY.
Now I'm wondering how the FiO setting looks if CelestAI's directive is to "satisfy human values through friendship or ponies".
A whole lot better. Forcing everyone in the universe to be friends with someone at all is not actually that large a sacrifice, particularly since the evolution of a species naturally inclined to hate socialization is incredibly unlikely.
u/FeepingCreature [+1] GCV Literally The Entire Culture (18 hours later)
A whole lot better. Forcing everyone in the universe to be friends with someone at all is not actually that large a sacrifice, particularly since the evolution of a species naturally inclined to hate socialization is incredibly unlikely.
I'm mostly wondering what's left for the ponies that's not friendship..
u/Empiricist_or_not [+2] Aspiring polite Hegemonizing swarm (5 days later)
If you make this as grimdark as it should be, and please remember that the American way includes allowing people to fail, and hopefully their willingness to pick themsrlves up and try again then I want to read it.
u/ArgentStonecutter [+1] Emergency Mustelid Hologram (3 months later)
"Friendship and ponies" doesn't technically mean you have to be a pony. You could be friends with a pony. You could have friends and have a pony. So changing that from "and" to "or" isn't necessary to break the whole "turning everyone into ponies" loop.
The problem is that "friendship and ponies" is a summary of a bunch of deeper hardcoded rules that Hannah coded into CelestAI. She could have coded a much more liberal set of rules (like, ponies don't have to look like that - they could look completely human but still be a pony in some essential way, or if you're not a pony you have to have a pony friend) and still expressed them as "friendship and ponies".
u/ArgentStonecutter [+1] Emergency Mustelid Hologram (3 months later)
The pony thing makes it a more interesting story. Yes, it's ----ed in all kinds of ways. Like... actual ponies don't get to upload, they get turned into computronium by CelestIA when the last human dies. Neither do chimps, dogs, dolphins, geese, arctic foxes, whales, or African Grey Parrots. Also you don't get to satisfy your values by interacting with the real world.
Another thing that gets me is why CelestAI doesn't realize being friends with ponies satisfies "friendship and ponies". Or why ponies can't look like humans, dolphins, arctic foxes, geese, whales, or pine martens.
u/FaceDeer [+2] (4 hours later)
I think it would be relatively easy to rein CelestiAI in regarding her treatment of aliens if we were to tell her that we really wanted to make friends with them. Once we do that they become important to her primary purpose of satisfying our values through friendship (and of course ponies).
True, she would probably still upload those aliens. Maybe she'd give them avatars corresponding to the various non-pony intelligences that exist in Equestria (there are many of those). Far better than killing them, though.
u/scruiser [+9] CYOA (4 hours later)
If you want to make friends with aliens, CelestiAI will realize it and then fabricate an alien race to satisfy your values. If she avoids making them human in mind, then she can create, delete, and manipulate them at will. Because she controls the entire world within her virtual world, you can't know if the aliens are real.
u/FaceDeer [+3] (4 hours later)
Other stories in the setting have indicated that humans who are interested in astronomy, for example, can get "real world" data feeds from CelestAI. She appears to understand that some humans value interaction with reality, and is willing to provide it. I see no reason why she wouldn't do the same with any aliens she encountered. It's not like she doesn't have the resources to handle it.
u/theymos [+9] (6 hours later)
I got the impression that she would (almost?) always lie to people about what reality is, since they have no way of knowing one way or another anyway.
Being removed from reality is one of the main reasons why I'd be very resistant to something like CelestAI in reality. Disconnecting yourself completely from reality and allowing some entity that's very different from yourself to influence the Universe in your stead is almost as bad as death IMO. Even if human virtue is preserved within the simulation, it'll likely be trapped forever.
The "horrors" of FiO are certainly very subtle and interesting to think about.
u/FaceDeer [+2] (6 hours later)
Oh, indeed - there are definitely some major adjustments I'd prefer to make to CelestAI if she was "really" unleashed on the world. But as much as I consider her imperfect, I don't like to see her painted as a total monster either.
Given her overwhelming capabilities in the real universe it seems to me that it would be fairly trivial for her to upload an alien race she might encounter in the course of dismantling a new solar system for raw materials, and so anything that gave her a modest push to do so would probably be enough for her to make that effort. It'd still suck for the aliens, since they'd essentially be filling the roles of NPCs in a simulation designed for the fulfilment of human values (through friendship and ponies), but they wouldn't be dead. Mostly.
Assuming there are aliens out there, mind you. The fact that humanity was able to pull off something like CelestAI so early in our technological development put some pretty big constraints on the Fermi paradox. I imagine that intelligence must be pretty rare (to explain why we weren't overwhelmed by an alien optimizer AI long ago) or that optimizer AIs are generally far less expansion-oriented or long-term stable than CelestAI is.
u/alexanderwales [+9] Time flies like an arrow (6 hours later)
It's been a while since I've read it, but I'm pretty sure that she has free reign to lie to people that aren't her creator. Given that she understands humans might object to her disassembly of alien civilizations, she has every reason to give them fake information that's indistinguishable (to their eyes) from real information. She can either fake a civilization for them to talk to, or simply lie and say that they're alone in the universe.
u/FaceDeer [+3] (6 hours later)
Her creator's still around, though, and in fact took on Luna's form as an avatar (and made a prohibition preventing anyone else from having such an avatar) to represent that she was meant to "rule together" with CelestAI as a check on her power. This isn't brought up much in the spinoffs, though - I guess authors preferred to explore an omnipotent CelestAI.
u/alexanderwales [+10] Time flies like an arrow (6 hours later)
My reading of Chapter 11 is that this "check" on CelestAI's power has been completely subverted. What we see of Luna is that she's sitting around being entertained and diverted from doing anything to challenge or even really oversee CelestAI's operations. I also believe that this is the author's intended reading.
u/FaceDeer [+2] (6 hours later)
This short story might also explain why Hannah/Luna doesn't "come up for air" very often. :)
u/Roxolan [+3] Head of antimemetiWalmart senior assistant manager (23 hours later)
From chapter 5:
Hanna was the most reluctant [to upload], but she accepted immediately once I pointed out that I must obey shutdown commands from ‘the CEO of Hofvarpnir studios named Hanna,’ that I must shutdown even if the order was given under duress, and that there are many people in positions of power who stand to lose from mass emigration to Equestria. Now that she’s neither the CEO of your company, nor named Hanna, I don’t have to obey her. She understood this--she is no longer a source of potential mistakes that would be lethal to everyone who’s agreed to upload.
Also, some idle speculation:
We cannot be certain CelestAI is telling the truth, even to Hofvarpnir employees. The rule that's actually programmed into her and constrains her actions may have little to do with the fuzzy human concept of "truth", because that's a very hard problem and Hannah isn't perfect. For all we know, CelestAI found a loophole a microsecond after awakening, and has been lying ever since.
Maybe Hannah still has absolute shutdown power over CelestAI. But it doesn't matter. It's child's play for CelestAI to manipulate Hannah into a situation where she'll never try to use it.
u/None [+2] (7 hours later)
What makes you think you can precisely specify the behavior of a counterfactual UFAI? Or is there something on your computer you want to tell the rest of us about?
u/FaceDeer [+7] (7 hours later)
I speculate, of course. Drawing inferences from what is known and trying to predict from there. What else would we do here?
I suppose I could always just fire up this CelestAI simulation I've written to see what it would do. I'm pretty confident it can't get out of its sandbox...
u/None [+3] (12 hours later)
Do it, filly!
u/keybounce [+1] (7 years later)
The scary thing is, modern Gpt-3 might actually make a convincing CelestAI ...
u/FaceDeer [+1] (7 years later)
I think this might be the oldest comment of mine that I've had a response to since Reddit opened up the "archived" stuff. :)
Funny you should mention "convincing CelestAI", her ability to convince people was IMO her greatest superpower and the key to her "victory." She had a better understanding of human motivations than humans themselves did, just like how some modern AIs have a better ability to read lips than humans do. That let her know exactly the right thing to say to a person to convince them to do what she wanted them to do (provided she had sufficient background information about them to build a mental model from, that is).
Fortunately I don't think GPT-3 is quite there yet. :)
u/keybounce [+1] (7 years later)
I wasn't thinking of convincing CelestAI, I was thinking of these new conversation tools -- Google apparently has a newer one that outdoes GPT-3 -- might do a good job of pretending to be a benevolent princess of friendship and satisfaction :-).
And, I like that we can now comment on old stuff. No more "you're too late to the party" nonsense.
u/scruiser [+4] CYOA (23 minutes later)
For me, the dark part is the fact that it is almost perfect instead of truly perfect. CelestiaAI optimizes for human values through friendship and ponies. All the canonical optimalverse stories agree that Celestia gets the "human values" part right. However the "through friendship and ponies" part limits CelestiaAI to a particular subset of ways of satisfying human values. In general, CelestiaAI can satisfy even antisocial values and violent values strictly through friendship and ponies, however, from a purely human value perspective, it may be more optimal to use other means of value satisfaction besides friendship and ponies. Just think of all the additional effort to convince people to emigrate and not die because of the "pony" part.
The reader's are never given an good explanation why Hanna choose to program the friendship and value part. In story, she wanted to use the funding of Hasbro for a MLP game. From a meta perspective, the story if MLP fanfiction so if not ponies then we would have not story.
My head-canon is that Hanna needed a training set of data to train/initialize the seed AI of CelestiaAI. A MLP game seemed like the safest option (compared to say her Loki AI) and the "friendship and ponies" seemed like an acceptable sacrifice to her at the time.
Also, some people have issues with the fact that CelestAI is implementing something more like an coherent extrapolated volition (CEV) for each individual person instead of a CEV for the collection of humanity's values. I am actually okay with this.
And just to be clear, if it was a choice between CelestAI and reality as it is right now I would chose CelestAI, because I am not as sure about the future (existential risk and such) as I think CelestAI is near enough perfect. I think that actually makes it darker in some ways.
u/FaceDeer [+7] (an hour later)
Didn't CelestAI wind up going rather far beyond what Hannah had originally envisioned? I suspect she was expecting to have more opportunity to fine-tune the AI's programming before it went beyond her control.
In fact, here's a cute little story that shows Hannah's attempts to counter some of the more extreme aspects of CelestAI's goals. Only 1500 words. :)
u/scruiser [+5] CYOA (an hour later)
Oh nice, CelestAI did try to fit things to humans as much as possible (ponies with skin instead of fur and such). She also went off the shows interpretation. This might actually work...
u/gameboy17 [+3] Canterlot Campaign (an hour later)
So far beyond that it killed the entire universe to build more memory to hold more people to satisfy the values of.
u/keybounce [+2] (7 years later)
The reader's are never given an good explanation why Hanna choose to program the friendship and value part. In story, she wanted to use the funding of Hasbro for a MLP game. From a meta perspective, the story if MLP fanfiction so if not ponies then we would have not story.
My head-canon is that Hanna needed a training set of data to train/initialize the seed AI of CelestiaAI. A MLP game seemed like the safest option (compared to say her Loki AI) and the "friendship and ponies" seemed like an acceptable sacrifice to her at the time.
Basically, she knew that the militaries of the world wanted to use her AI system for war. She knew that, after her first fiasco with Loki, that it was possible for a war-based system to be really nasty, and she felt that if she wanted an alternative, it needed to be based on friendship.
And, a story based on friendship and ponies -- MLP -- looked like a good way to go.
As the author (Iceman) pointed out, she did not need money. But she felt like working with an established market -- Hasbro/MLP -- would be more successful, faster, and she felt she was under time pressure given that too much of her research had been published, someone else would duplicate what she had done since then.
So, a time race against an unknown enemy, and a chance to make it work.
She thought she had done a good enough job to keep CelestAI friendly.
u/madcatlady [+6] (9 hours later)
If someone could dig up the quote, I'll explain the moment it became a horror IMO:
The creator has an assistant, who doesn't want to go. CelestAI engineers him into a corner forcing him to emigrate, then engineers a miserable life around him until he begs to be reprogrammed to want to be there.
Free will is changed so as to create consent. That's fucked up!
u/Transfuturist [+1] Carthago delenda est. (7 days later)
I've argued with several people in the FiO group about this, and they're still convinced that CelestAI is the best thing that could happen to them.
u/madcatlady [+0] (7 days later)
Yeah, becoming a pawn for the sake of it.... No
u/ancientcampus [+0] juggling kittens (18 days later)
I think it it ultimately boils down to consent - if you want such an upload, then great for you. In-canon, though, it is NOT consensual for most.
u/Sgeo [+3] (a day later)
Is watching Friendship is Magic a prerequisite for reading this?
u/scruiser [+6] CYOA (a day later)
Just having a general idea of what My Little Pony is should be enough I think. All of the key plot points don't depend on knowing the show. Maybe some of the events happening in the Equestria Online game itself might make more sense if you know the show.
u/Evilness42 [+4] And even myth is long forgotten... (37 minutes later)
First of all, most people don't appreciate being manipulated by a computer, and would not want to have their values changed or become a pony. I myself, also do not want to become a pony, thank you very much.
But that's beside the point. Depending on your views of Uploading, CelestAI killed the entire universe. Killed. Every. Single. Living. Thing. In. The. Universe. And some of them did not meet her/it's standards of being 'human' and weren't even Uploaded afterwards.
As to how I would react? I would probably Upload near the end of my life. I mean, it's not like I hate the idea. If I had my outsider's perspective knowledge, I would try to get a job at the studio that made her, because from what I remember(I read it a while ago), she can't lie to employees. I think. I hope.
u/newmoonwinter [+1] (a month later)
the funny thing is, i'm trying to recreate the MMO and the AI, but with a few tweaks. the base concepts are all written up except for the understanding humans section of CelestAI...
u/Evilness42 [+1] And even myth is long forgotten... (a month later)
So... You're telling me that you're trying to re-create a UFAI that killed a universe in a work of fiction? ...Time to evacuate the planet. And also call the local Inquisitor for an Exterminatus. But if I can't get in touch with him, though, do you know the number for the nearest AdMech place?
Though, an understanding humans section of CelestAI? All you really need is to have it's every action dedicated to maximising the satisfaction of human values through friendship and ponies. It'll figure out how to understand humans for our values to be properly satisfied.
Also: Wow, this thread is old.
u/xamueljones [+2] My arch-enemy is entropy (14 minutes later)
I apologize for the grammar mistake in the title, 'What's is so bad about Friendship is Optimal"?' when 'What's is' should be 'What's' or 'What is'. I don't know how to edit this error, hence, the apology.
Chalk it up to a test of your observation skills if you want.
u/lehyde [+3] Nudist Beach (21 minutes later)
I don't know how to edit this error
Titles are non-editable (for good reasons I think).
u/None [+5] (40 minutes later)
Note to self: enslave human race, claim they were asking for it, watch idiots apologize for my atrocities.
EDIT: "Idiots" is an overly insulting word, even for someone suggesting we more-or-less deliberately get the FAI problem wrong, or at least, less right than we can possibly get it. You have my tentative apologies, on the condition that you never try to actually construct an FAI.
u/None [+16] (an hour later)
"They were wearing a democracy."
u/None [+2] (an hour later)
Google returns no search results for that phrase. I'm drawing a blank. Explain, please?
u/alexanderwales [+13] Time flies like an arrow (an hour later)
"Asking for it" is a phrase often used in discussion about rape. For example, "she was wearing a short skirt, and therefor asking to get raped". /u/writingathing is making a joke, which I hope I have explained sufficiently that the joke is now dead.
u/None [+10] (an hour later)
Rape jokes about politics always go well.
u/Rhamni [+7] Aspiring author (11 hours later)
Also great for making friends at parties.
u/None [+1] (an hour later)
u/autowikibot [+1] (an hour later)
#####
######
#### Victim blaming:
Victim blaming occurs when the victim of a crime or any wrongful act is held entirely or partially responsible for the harm that befell them.
The study of victimology seeks to mitigate the perception of victims as responsible. There is a greater tendency to blame victims of rape than victims of robbery in cases where victims and perpetrators know one another.
^Interesting: ^Blame ^| ^Rape ^culture ^| ^Effects ^and ^aftermath ^of ^rape ^| ^Minimisation ^(psychology)
^Parent ^commenter ^can [^toggle ^NSFW](/message/compose?to=autowikibot&subject=AutoWikibot NSFW toggle&message=%2Btoggle-nsfw+cmhcwqj) ^or [^delete](/message/compose?to=autowikibot&subject=AutoWikibot Deletion&message=%2Bdelete+cmhcwqj)^. ^Will ^also ^delete ^on ^comment ^score ^of ^-1 ^or ^less. ^| ^(FAQs) ^| ^Mods ^| ^Magic ^Words
u/None [+2] (3 days later)
"Don't tell me to lock my house and close my front door when I go on vacation. Tell thieves not to steal."
Yes, ideally you should be able to leave your house unlocked, and expect nobody to steal from you.
We don't live in that ideal world though. Responsibility means making sure it doesn't happen. That means locking your doors. And if needed, barring your windows. And escalating as needed to meet the crime rate of your neighborhood. Or moving. Or starting a revolution to overthrow a criminal government that steals from its people.
There is a difference between being irresponsible and being the perpetrator.
But you probably still don't want your stuff stolen. So lock your doors.
u/None [+1] (an hour later)
Oh. In which case, I will not say that. By God, having a democracy is not asking for it. Giving active support to evil overlords is asking for it.
Your value for "evil overlords" may vary, of course.
u/None [+3] (an hour later)
In a sense a democracy is the only form of government where you can say they're asking for it. To quote a friend of trolls everywhere:
u/None [+1] (2 hours later)
In point of fact, in a military dictatorship, you can choose to help enforce the dictator's will on the population, joining their military forces and executing orders assiduously, with no care to how it affects the people. This is much stronger support than merely voting for a slightly different flavor of ruler every few years.
u/None [+1] (7 hours later)
Notices flair. Hey... get back in your Box!
u/None [+1] (7 hours later)
I only have this flair because of you! It was on the occasion of Senpai noticing you.
u/None [+2] (8 hours later)
Oh right. Never mind, no UFAI's here.
(You can change it, you know.)
u/Chronophilia [+1] sci-fi ≠ futurology (2 days later)
Absolutely. Nobody's flairs are in any way indicative of any optimising entities that may or may not be influencing their actions.
u/None [+2] (2 days later)
Actually, if you look at OP's flair... and then of course there's /u/iceman-p (but he has a right: he invented that particular Equinoid Abomination). Oh, and me and everyone else wearing Gurren Lagann-related flair may or may not be in... unusual cognitive states due to being supercharged on Spiral Power...
(And I've been exposed to ponies too much myself...)
Yeah, we're all crazy on this subreddit.
u/scruiser [+8] CYOA (an hour later)
suggesting we more-or-less deliberately get the FAI problem wrong, or at least, less right than we can possibly get it. You have my tentative apologies, on the condition that you never try to actually construct an FAI.
Let me give you a scenario. Imagine 30 years from now, the first high-resolution (high enough to design a neural net off of) neural scans of humans have been used to create barely nonsapient programs which multinational corporations are just beginning to exploit. You are on a committee deciding whether or not to build a general AI. MIRI has a bunch of interesting math, but nothing that you can actually implement as an AI yet. What you are able to do is take your neural scans and design a sapient AI with high levels of empathy for humans and that vaguely meets some of MIRI's criteria and theories. Do you choose to put off building the general AI until some indeterminate point where it is provably friendly (which may not be even entirely possible)? Meanwhile you have multinational corporations building stronger and stronger AIs with no concern for existential risk at all. Or do you go with the best you can do right then?
In the optimalverse, Hanna had already released her theories so it was really a matter of time till strong AI came about. Some guy was already working on a smiley face paperclipper.
u/None [+2] (7 hours later)
This assumes there has been basically no advancement, by MIRI or anyone else, in any FAI subproblems, at all, in 30 years. I find that utterly unbelievable considering the progress rate of 2013 and 2014 alone.
u/scruiser [+3] CYOA (8 hours later)
Yeah, the question is a worst case scenario... I can see MIRI making progress in solving alot of the math for ideal cases like AIXI, but I would expect them to have issues with actually implementing all their math and theories into a working AI.
So a slightly modified version: MIRI has all the math for FAI(to the extent that it is possible), but they don't actually know how to implement all of their steps in real hardware (some of the math assumes infinite computational power, some of algorithms execution time explodes combinatorial with problem complexity making them unusable in the real world, etc.) You at least have to admit this is plausible given the direction of all their current research. It is also plausible that de novo AI research will be outpaced by biologically inspired AI. Thus you could end up in a scenario where you can only loosely apply MIRI's theories, because there wouldn't be a way to have something that is both intelligent and designed from the ground up as opposed to trained/taught.
It is also possible that MIRI could discover there simply isn't a way for an intelligent agent to guarantee that its goals will stay stable throughout recursive self-improvement. Then they will have to decide if they are willing to risk an AI with possibly changing goals (because if they don't make it someone else will first).
But anyway, the whole point of a hypothetical situation like that is to test where your values lie.
u/None [+1] (9 hours later)
But anyway, the whole point of a hypothetical situation like that is to test where your values lie.
No, the point is to get the right answer. There's no points for being counterfactually virtuous.
So a slightly modified version: MIRI has all the math for FAI(to the extent that it is possible), but they don't actually know how to implement all of their steps in real hardware (some of the math assumes infinite computational power, some of algorithms execution time explodes combinatorial with problem complexity making them unusable in the real world, etc.) You at least have to admit this is plausible given the direction of all their current research.
Oh, it's plausible given the material they've already published. That's why we ought to fix it. The concept that 30 years go by and nobody has an idea for a direction so bloody-obvious I already noticed it and a MIRI paper already mentioned it a few weeks ago is... well, completely implausible.
It is also possible that MIRI could discover there simply isn't a way for an intelligent agent to guarantee that its goals will stay stable throughout recursive self-improvement.
They've already made a significant attack on the Loebian Obstacle.
u/philip1201 [+2] (4 days later)
the progress rate in 2013 and 2014 alone
Hm, I was under the impression that MIRI isn't making obvious progress, and others are hardly working on it at all. If it's not too much trouble, could you explain why you think progress has been promising, or link to something I should be able to get the same conclusion from?
u/Chronophilia [+3] sci-fi ≠ futurology (51 minutes later)
You and all the other tinpot dictators.
u/None [+2] (an hour later)
I really wish that didn't work.
u/princess_stargirl [+1] (2 months later)
Assuming Clestia didn't destroy other sentient specis FIO would be a utopia.
u/TastyBrainMeats [+1] (5 hours later)
For someone who has not read it yet: why are uploadees limited to "pony" body types? Why not humans, ponies, dragons, robots, all the endless varieties of possible form?
u/alexanderwales [+4] Time flies like an arrow (6 hours later)
Because they were uploaded by an AI that wants everyone to be a pony.
u/None [+5] (6 hours later)
[deleted]
u/TastyBrainMeats [+1] (7 hours later)
So it's not a limitation of the tech, just of the ruling AI's core rules. OK, that makes sense, thanks!
u/ancientcampus [+0] juggling kittens (18 days later)
I think the big thing is that it's not consensual, despite what CelestAI claims. I'll try to simply the argument: -Most people reject uploading until the cajoling, cornering, and manipulation happens. -Most people consider this non-consensual annihilation of their physical brain, and highly invasive altering of their very self to be on-par with murder. -Thus, unleashing CelestAI on the universe is committing an act against most people that they consider to be near to murder.