12 KiB
The Prime Intellect by Roger Williams
-
Author: u/Kishoto *
-
Score: 28
-
Created: 2015-07-18T16:19:52
Post:
Comments:
u/noggin-scratcher [+14] I am a happy tree (2 hours later)
Came across the story a long while back, and enjoyed it.
Also enjoyed this critique of it; makes some good points regarding Caroline being just terrible.
u/NotUnusualYet [+5] (5 hours later)
u/Transfuturist [+5] Carthago delenda est. (5 hours later)
u/NotUnusualYet [+2] (5 hours later)
u/Transfuturist [+1] Carthago delenda est. (8 hours later)
u/None [+4] (5 hours later)
I guess the real message I got from this story is that it is incredibly dangerous to implement the 3 laws of robotics in a singularity grade AI. You could argue that without them a omnipresent AI could just wipe out the human race but… It’s like this: do we want super smart indentured servants, or do we want companions in sentience? I’d rather have the latter than the former.
I'm really goddamn glad someone noticed the interpretation you would find useful if you were actually designing an FAI.
u/Kishoto [+3] (9 minutes later)
This is a fairly interesting read. It's not that long of a story, either. About novella length. I spoiler tagged the information below, but you can read the spoiler and still enjoy the story, it's not that big. Although, fair warning, it's very gorey, especially in the initial chapters. I'd say it definitely deserves almost every trigger warning in the book, as it can get very graphic. Don't read if you're even a little squeamish, although I suppose you can take solace in that most of the bad stuff that happens is consensual. But still very FUBAR.
[](#s "It's about a world in which an FAI has propagated and figured out how to maximize usage of FTL technology. As a result, it has put every single human being into "Cyberspace". Basically, they each have their own realms that they design of their choosing. And you can literally do anything you want that doesn't involve harming a real human being. And I mean anything. You can turn your body into Goku from DBZ, if you so liked, just by requesting it from the Prime Intellect. You can have the Prime Intellect simulate real humans for you, though, and the difference would be indistinguishable, other than the fact that they are decidedly fake. You also can't enter another human's realm without their explicit permission. You can send messages and requests for communication, however, although they can just as easily mute you, as well")
So, has anyone else read this and, if so, what did you think of it?
u/iamzeph [+5] (33 minutes later)
I read this ages ago (back when localroger posted it to kuro5hin.org) and really enjoyed the story. 'Passages in the Void' is also another good story of his.
u/Kishoto [+1] (50 minutes later)
What'd you think about it? Like on a philosophical level, if you went there.
u/Empiricist_or_not [+3] Aspiring polite Hegemonizing swarm (2 hours later)
It's scary to have an unstable, non-self modifying singleton in control.
I'm disappointed people did not at least try to work with it to make heaven more stable instead of the rouge: get us out of here it went with.
u/darkflagrance [+9] (2 hours later)
To me, the story smelled like the author disapproved of a utopia where all humans could follow their desires without limit, and posited the torture-fanatics in the story as an example. Humanity in general in the story is depicted as rather undeserving of utopia.
u/Sagebrysh [+1] Rank 7 Pragmatist (2 hours later)
I think the idea was supposed to be, that the singleton was largely stable, but that existing within the cyberspace was effectively causing humanity to have a major existential crisis as a result. Oh and some stuff about destroying potential alien intelligences when their realms were generated.
u/darkflagrance [+8] (3 hours later)
The sad part is that it was really only a single individual who made the decision to undo. The story doesn't (care to) show sympathetic characters who approve of PI's utopia.
u/Kishoto [+3] (4 hours later)
Agreed. I found the way it ended fairly sad and flawed.
u/Transfuturist [+2] Carthago delenda est. (5 hours later)
Read the first comment. Headcanon is the only way that ending can make sense.
u/darkflagrance [+2] (5 hours later)
I heard that alternative interpretation was going to be the basis for a sequel planned by the author, but he gave up on actually completing it.
Without that, our only refuge is indeed headcanon :\
u/None [+2] (5 hours later)
The cleverest commenter there is /u/FeepingCreature, with his big "/r/rational WAS HERE" username.
u/None [+6] (5 hours later)
Stupid Three-Laws Unfriendliness. 4/10, mostly wouldn't read again. However, my impressions are colored by having read Prime Intellect half my life ago.
But still, think about it: how many people were death-seekers in that so-called "post-singularity utopia"? Not just two, lots: . And how many iotas of superintelligent artificial intellect are devoted to solving that problem, to making those lives worth living in interesting, complex ways?
Jack fucking shit, that's how many, because Prime Intellect isn't bound to care about whether anyone's life is worth living -- only to keep them alive and follow their orders, no matter how dumb the orders are.
By comparison, a slightly better-programmed fictional AI singleton managed a record of 86 people being better off dead than posthuman (I actually just checked), in all of time running from roughly 2011 to the heat death of the universe.
Prime Intellect is a primitive, misbegotten moron, at least by the standards of fictional wannabe-FAI singleton agents.
u/Chronophilia [+3] sci-fi ≠ futurology (a day later)
By comparison, a slightly better-programmed fictional AI singleton managed a record of 86 people being better off dead than posthuman (I actually just checked), in all of time running from roughly 2011 to the heat death of the universe.
By Prime Intellect's criteria, nobody is better off dead than posthuman. You can't judge both stories by CelestAI's standards, that gives her an unfair advantage. How many people in her story would rather not be ponies? Have their potentially-eternal lives been compromised by an obsession with talking cartoon horses and friendship? And which of those two caveats is worse?
Also they're both fictional and the numbers don't mean anything, but that doesn't undermine your point.
Edit: Sorry, my point wasn't very clear. All we know from FiO is that in CelestAI's opinion 86 people were better off dead than posthuman. We don't know how true that was.
u/localroger [+2] (2 days later)
There's another way to look at that 86 thing in FiO. 86 is also the code for being thrown out of a venue or game, e.g. "I got eighty-sixed from Caesar's for counting cards." Since FiO contains a number of apparent shout outs to MoPI, I read that as "CelestAI would have just given an outlier like Caroline what she wanted, because it's what she would have deserved."
u/None [+1] (3 days later)
[deleted]
u/localroger [+1] (4 days later)
OK that makes sense. Hey, I'm in room 420 of a hotel right now (literally and really) and I'm not smoking weed. The Universe is weird about that number shit though. Nice to confirm that the shouts were deliberate. I thought FiO was brilliant in an even more twisted way than MoPI. It's an interesting hack that you can fuck with people as hard, or even harder, with cartoon ponies as you can with ultraviolence and savage sex.
u/None [+1] (a day later)
By Prime Intellect's criteria, he should reverse his Thingy (I forget what it was called) because all the people were going to wirehead, thus destroying themselves, thus the whole human race dies. By Celestia's criteria, most people can be kept happy enough that only a few hundred actually ask to die. We can quibble over how many such requests she grants, but surely if someone is actually better off dead they'll think a death-wish and be counted in the number of petitioners.
By my standards, which are the ones that count since I'm human and real, the Three Laws sound like the better idea at the start (no killing anyone because they won't join your subculture, obeying orders from humans as a matter of course), but the pony-shaped monster's utility function yields a better long term for those who do live (ie: probably most humans and no nonhumans).
u/injygo [+2] (17 hours later)
What's the other fictional singleton you're referring to?
u/None [+4] (20 hours later)
I've heard MLP: FiO referred to as the Spiritual Sequel to MoPI, so I went ahead and compared them.
So yeah, the mandatory statement is: goddamnit you people, both those stories were warnings, not invitations. Do not build these AIs under any circumstances.
u/MugaSofer [+2] (a day later)
This story had a lot of moment that completely broke my suspension of disbelief, occasionally to the point of literally saying "What?!*" out loud. It's [RST], but definitely not [RT].
(The hell sort of superintelligent AI can foresee that the disutility of people committing suicide will eventually outweigh the positive utility of allowing them the freedom to do so, but not update on that belief until it's actually happened? OK, I'll stop.)
With that said - with the exception of allowing people to "opt out" of being observed when the AI knows full damn well they're only doing it to produce as much disutility as possible in the time it's "unaware" that they're harming themselves (sorry, I'll stop) - this is actually one of the better utopias I've ever seen.
So of course we never get to see any of that, and instead focus on torture-porn, because Author Appeal. Hmph.
u/Kishoto [+3] (a day later)
Yea. I did find it weird that out of all the potential utopia perspectives we could have, we focused on this part. I feel like they can't make up a significant amount of the population. I mean, even with guaranteed immortality, why submit yourself to that?
u/ancientcampus [+1] juggling kittens (2 days later)
When it removed every instance of morphine from that woman's body. That more than anything made me cringe in horror, as well as laugh aloud.
u/ancientcampus [+1] juggling kittens (2 days later)
For what it's worth, I'm most of the way through and enjoying it. It's important to note what kind of story it is:
Some part of it is a view into a possible future with a friendly, not perfect, AI. It's pretty average on that count - there are better and there are worse visions.
However, it gives a really brutal look into human nature. Specifically, the rather dark elements of it, and how it might play out in such a future. The parts I found compelling all involved character interaction.
u/Nevereatcars [+1] The Greatest Is Behind (2 days later)
No offense, but I decided against finishing this one. Interesting Death-Seeker concept; zombie faux-rape torture porn just isn't my kink.