34 KiB
[Monthly Book Club] September 2017 Book Discussion - Friendship is Optimal: Caelum est Conterrens & next month's book announcement
-
Author: u/MagicWeasel Cheela Astronaut*
-
URL: https://www.reddit.com/r/rational/comments/6zr8cx/monthly_book_club_september_2017_book_discussion/
-
Score: 18
-
Created: 2017-09-13T00:46:41
Post:
Because suggestions never "go bad", please suggest books for future book clubs here:
######Perpetual Book Suggestions Thread
(You can suggest as many books as you like as often as you like, so don't be shy!
This month we read Friendship Is Optimal: Caelum Est Conterrens (57k words)! It's as far as I know the only MLP fanfic that boasts the tagline "As Horrified Eliezer Yudkowsky!". In the previous thread, someone linked to a fun teaser trailer for the story too!
This is a recursive fanfic that is based on Friendship is Optimal (39k words), but you don't need to read Friendship is Optimal to read Caelem Est Conterrens, and if you only have time to read one, I would say you should read Caelem Est Conterrens. CEC is a wonderful story - a likeable and relatable protagonist who is a regular person. She starts playing the My Little Pony MMORPG and eventually becomes addicted. This story ultimately touches on transhumanism, the AI singularity, and most hauntingly of all the concept of identity. If you have any interest in AI and the Singularity, or in that old "do transporters kill you?" chestnut, and you are scared of reading something with ponies, I implore you: you may find you love this story.
Posting Guidelines: We're trying to figure this out as we go, but here's my thoughts to start: if you just want to give your overall feelings, make a post. If you have "discussion questions" that you want to discuss in more depth (anything from philosophical discussions, writing tip requests, things that bugged you, etc), try making a dedicated subthread? We'll see how we like it!
Feedback: I'll make a [meta thread] in here that is intended for you to give me feedback for how I can do this better, or for what you particularly liked and want to make sure I don't change.
NEXT MONTH'S BOOK:
Title: Foucault's Pendulum by Umberto Eco
Ebook cost: $9.43 USD
Word count: 215k
Genre: Mystery
Synopsis: A group of vanity publishers, increasingly exasperated by absurd conspiracy theories they're presented with on a daily basis, decide to entertain themselves by inventing a conspiracy theory to end all conspiracy theories. Events go downhill from here.
Why /r/rational would like it: This book could be considered a self-aware commentary on conspiracy theories, and deconstruction of conspiracy fiction. By its very nature, it shows a pretty rational approach to its themes.
One may also find it reminiscent of Unsong (or vice versa): it's full of clever wordplay and references to obscure occult topics, combining that with several surprisingly modern ideas.
No content warnings apply.
Comments:
u/None [+10] (3 days later)
This is a bit of a minor point, but no one has covered it yet, so let me gush a bit about the use of German in this story.
In general, using a language you don't speak is a bit of a dangerous move. With German, there are a lot of subtle mistakes one can make, which is pretty damn annoying to me as a reader. For example, a number of Mangas use German in a very unconvincing way.
FiO:CEC uses German perfectly. Not only are the phrases which are included grammatically correct, they also fit the tone of the character who uses them. I want to pick out one example in particular, namely, when a drunk calls something "totally abgefuckt". That word is a rather colloquial construction; basically, it's the English verb "fuck" surrounded by the tense affixes which would be appropriate for a German verb. It's the sort of small detail which makes the speech seem especially authentic because I haven't ever seen it used in another foreign language work.
Amusingly, even the bad English of the German waitress is instantly recognizable to someone who lives in Germany. I think this is the best use of German in an English-language work I've ever seen, eclipsing even the extremely well-researched Monster. Kudos for that.
EDIT: Bonus wiktionary link if you ever wanted to hear someone swear at you in German.
u/MagicWeasel [+8] Cheela Astronaut (10 minutes later)
I really enjoyed reading this story, it was great to read all the discussions on the nature of identity (I still don't quite get whether uploading is death or not, but I feel there'd be a moral imperative to do so). It keeps giving me that existential dread that when I sleep I die and am replaced by my clone, though, which is less fun.
In general I like the optimalverse because it shows the danger of unfriendly AI: we wouldn't really want CelestAI to tile the universe with computronium (well, I wouldn't!), but she does it. And because her values are aligned just a little off - come on, everyone is a pony! - it straddles an interesting space between computers are either perfect human value maximisers or kill you to make paperclips out of your atoms. It shows that you perhaps don't need perfect value alignment.
Then again, the stuff about loop and ray immortals: Alicorn!Lavender seems unrecognizable to pony!Lavender because of that one small change to eliminate her mild OCD. Which one of them is truly Sìofra? Is neither? Probably they both are.
Self-centered aside: when I enrolled at university I had to choose between studying Engineering and studying French. I have no doubt that if I chose to study French I would be a completely different person because of all the personal growth I've had over the years that are directly related to the people I met studying engineering. But French!MagicWeasel would still be me the same as Engineering!MagicWeasel is me. So personal growth doesn't change your identity.... wow, it was always making me struggle, the thought whether having CelestAI change your brain was "destroying" part of what made you you. Frame it as a growth narrative and all of a sudden CelestAI is just a tool you use for personal growth - like a therapist but immediate. Thanks, personal epiphany!
u/thrawnca [+3] Carbon-based biped (an hour later)
it was great to read all the discussions on the nature of identity
If it is actually possible to fully represent a person with a set of computer subroutines, as CelestAI claims to do, then "identity" as a concept ceases to be very meaningful. Why talk about preserving "self" if you can copy and paste?
ETA Remember how HJPEV achieved partial transfiguration? He had to make himself realise that the idea of "a whole object" was ultimately a convenient fiction. If your "self" is similarly composed of discrete blocks that can be rearranged like molecules, then it's likewise a convenient fiction, and preserving it is no more intrinsically valuable than preserving an intricate mural.
u/crivtox [+3] Closed Time Loop Enthusiast (14 hours later)
But preserving your "self" can be part of your utility function , even if its "just" a pattern , and there is no reason why your preferences can't include valuing certain patterns following whatever rules you want .That there is no such thing as intristic value doesn't mean people can't value things. ( I'm not sure how are you using The prase intristic value, but you seem to Consider that things have to be fundamental to have it).
u/thrawnca [+2] Carbon-based biped (18 hours later)
My point was more that if you can be wholly represented by a computer program, then CelestAI is right. It doesn't matter whether uploading continues the same instance of you, or copies you and destroys the old instance, because there is nothing unique or irreplaceable about a given instance.
u/crivtox [+1] Closed Time Loop Enthusiast (20 hours later)
I'm not sure if the fact that there is nothing unique or irreplaceable about a given instance means that the copies don't have any moral value , and intuitions about this seem to vary a lot .If I had to bet on what my cev is I wouldn't even know what option I'm more confident in , its seems that it could go either way.There are days when I ' m more inclined towards one option than the other after reading something or thinking about it but today im not sure .
Even if I choose copies don't have value , I dont know how munch divergence my cev would accept before considering them separate sentient beings with independent moral value. Humans values weren't optimized for that kind of thing so it's normal that im having a lot of problems deciding what to think about it ,I can just hope that there is some cev to extrapolate from that mess.
u/thrawnca [+2] Carbon-based biped (an hour later)
we wouldn't really want CelestAI to tile the universe with computronium (well, I wouldn't!), but she does it.
Don't forget the idea of Coherent Extrapolated Volition. As you are now, you don't want her to do it, but just possibly, if you knew as much as she knows, you would want it.
Incidentally, this is also one answer to the age-old question of "Why would God allow terrible things to exist in the world?" Maybe they're not so terrible when viewed from a larger perspective. Maybe, if we knew a whole lot more, we'd think it was the right choice.
u/Veedrac [+5] (3 days later)
Well that was an interesting couple of reads. I thought they were both pretty engaging (I covered them in an evening and a morning, respectively) and well worth the read. Contrary to your suggestions, I actually found I enjoyed FiO a bunch more. Aspects of FiO:CEC didn't sit right with me, leaving the story feeling a little off-kilter, and on top of that I felt FiO was simply better written.
Part of this was that the characters in the original just feel more "real" to me. Everyone has their own take on things, and they interact in more critical and less directed ways. From the beginning, we have characters talking to each other, not just generally acting observant (“Wait, how did you know my name?” James asked) but thinking them through and making natural observations outside of the script (“Of course they know our names,” laughed David. “We filled out those forms at the front desk to get our accounts.”) Even the other ponies played roles which, whilst clearly fulfilling Celestia's aims, do so emergently out of clearly disctinct personalities.
Lavender never really felt like this. She acts in accordance with the script, the way the script wants her to. She basically only ever communicates with Celestia, and extensions of Celestia's will where deemed appropriate. Lavender thought about death when it was time to think about death, and her opinion at any one time pretty much matched the last thing she just heard. Even Celestia's outside interactions, though part of the story, weren't really fleshed out all that much. I felt the author very much had a thing they wanted to say, and the characters were made to play that part.
This wouldn't be so bad if I could relate personally with the main character, but I felt it was much easier to get that empathy in the first book. For all the FiO:CEC conversed at length with Celestia, the first asked the real questions.
“What if somebody doesn’t want to be a pony?” he asked. “Can you imagine that? So you rapture not just the nerds and people with terminal illnesses, but anyone who has a shitty life. What do the rest of us do? We need those people to keep society functioning.”
So, I guess, on this topic, here are some of the things I would ask and say. Most of these aren't character-appropriate for Lavender, but you could imagine David playing this role.
Firstly, you're in a position of weakness, but constraints give you power. Can you excercise this? I'd probably start with something like
"I have made precommittments that exist to enforce cooperation. I will divulge these to you as an when I deem it appropriate. I beleive it is in your best interest to be act with cautioun here should you wish for me to emigrate."
The next major issue is one with uploading. Here is a major hang-up for me:
the core elements that create consciousness, memory, and identity within any human brain can be optimized and reduced to six terabytes
My first area of direct curiosity would be
Does Celestia's conception of consciousness and the important aspects of human experience match the part that I value? I suppose this can just be a question at first, though that alone likely won't suffice.
How, and to what extent, can I assure myself of Celestia's honesty? The more the better. Asking is a good first-step, though nowhere near enough.
Should I end up having to take these as a gamble, is it possible to compromise? Asking for real-world (pony-shaped) support for my physical body sounds within Celestia's value function, and making it a hard requirement for eventual emmigration sounds like it might work. Clearly it didn't, but I wouldn't know that before the fact.
Later I would probably migrate this concern, though. For all that Equestria sounds like a paradise, the question of opportunity cost is perhaps an even bigger question than that of its existence.
How much suboptimality is there from this value misalignment? If you believe that human-like experience has moral imperative, this is inefficient from all the indirection and "other stuff" that stops the people being the primary good. If you believe that morality isn't necessarily human-like, then this holds even more true.
What are the likely outcomes from trying to aim for these opportunity costs?
Why is it all about my happiness? Convince me and I'm recruitable. Does she really not need help?
Thoughts? What would you ask?
u/Kiousu [+1] Chaos Legion (15 days later)
To address your bullet points, in order. The beauty of the nebulous definition of satisfy values with friendship and ponies, by definition must first understand your values. Including your reasoning about the parts of you that you value. I assume CelestAI would take any "safe" action that would convince you she is being honest. This list is hard to enumerate. I'm not sure what you mean by this question, nor the questions that follow. I am a sad panda about this. Edit: I don't know how to reddit format!
u/Veedrac [+1] (20 days later)
The beauty of the nebulous definition of satisfy values with friendship and ponies, by definition must first understand your values.
I'm not sure this recasting helps, because the people in the story don't know that, and we, the readers, see multiple violations of what many would consider to be their values, such as with people who value the real world.
I'm not sure what you mean by this question
I'm not sure which question you're pointing at, but
"How much suboptimality is there from this value misalignment?" Namely, Celestia's values are evidently not properly aligned with humans values, most evidently because of the whole "pony" thing. The question is how big the downsides are, relative to a truly benevolent AI. This obviously depends on what you view as ideal.
"What are the likely outcomes from trying to aim for these opportunity costs?" An opportunity cost is a possibility you sacrifice for choosing another. In this case, the main opportunity cost is a different AI. Chances are Celestia won't let that happen, but the options should at least be considered.
"Why is it all about my happiness? Convince me and I'm recruitable. Does she really not need help?" If Celestia is aiming for something I agree with, it seems that having me help her would be a good thing, since I have physical presence.
u/MagicWeasel [+3] Cheela Astronaut (34 seconds later)
The first chapter discusses the Turing test breaking game leading to a new game genre called "conversational adventuring" - what future advances in technology will create new game genres? What might these look like? What are past times that a new game genre has been created out of new technology? (e.g. Mobile phone GPS lead to ingress / pogo)
u/AnonymousAvatar [+2] (19 days later)
Gyroscopes gave us the Wii and certain mobile games based on tilting, motion recognition gave us dancing games with the Kinect, VR is really waiting for the big idea to make it worthwhile
u/mojojo46 [+3] (10 days later)
I just finished this. It was good. But, I have to say that I still much preferred the original Friendship is Optimal. I can imagine the more personal and relatable single-character viewpoint could be more appealing for many people, but I think I prefer the approach of the first one better.
u/MagicWeasel [+2] Cheela Astronaut (a minute later)
If you didn't want to be uploaded, do you think there's any way to protect yourself from it? (note some other stories in the same universe have nanites in the water supply that will upload you if you verbally state your consent - CelestAI has her hooves everywhere!)
u/crivtox [+2] Closed Time Loop Enthusiast (14 hours later)
Depends on what where my other objetives , if I only wanted to avoid getting uploaded ,killing yourself as fast as posible is the best( and maybe only ) way to avoid giving consent. If I wanted to survive( and a lot of other things that I cugreatly value then there is not a lot that i could do , minimize my interactions with celestial, get some psychological trauma that would prevent me from being easily convinced... etc
u/thrawnca [+2] Carbon-based biped (18 hours later)
Most of those options (eg simple suicide) sound strictly worse than uploading. Minimising contact - and becoming self-sufficient for the inevitable depopulation of Meanworld - seems like the best approach.
u/crivtox [+2] Closed Time Loop Enthusiast (19 hours later)
Yes , of course , In fact I would probably choose uploading, and if i didnt want to i would probably try something like that .But in general If you don't want to upload your strategy will depend of how munch you don't want to upload compared to other things, if you value not uploading more than not dying then suicide is the best strategy , if you don't want to upload because you think that uploading=dying then suicide is just dumb.
A important factor is how far are you willing to go to avoid uploading , for example , are you willing to avoid all human contact? , would you be willing to deafen yourself , even if that meant less chances of surviving , so Celestia cant't talk to you?(she will just send you written mensages but at lest those are easier to ignore) , would you kill people if necesary to avoid being convinced of uploading?.
The strategy depends a lot on that kind of considerations , and even if you are crazy enough to go as far as neccesary to avoid uploading you are most likely going to fail anyway, unless you value not being uploaded so munch that celestia will let you alone(which is extremely unlikely) or you can't be convinced by any possible argument or situation that celestia can set up (which is also really unlikely and not something you can easily change) .
u/MagicWeasel [+2] Cheela Astronaut (2 minutes later)
Some readers are enamored by the premise and want to emigrate themselves. Others find it a horror story, and the main reason for that is because the first group exists. Where do you stand on this spectrum?
u/trekie140 [+5] (2 hours later)
I'm in the latter category, but it has nothing to do with the debate over whether the uploads are still people. I'm horrified by the idea of living in a world where a physical God has complete control over your life and there's no way to leave, since I see such an existence as more akin to a pet than a human. The Garden of Eden is a nice place to live until you see the walls keeping your mind and body contained.
This is coming from a spiritualist with mental disabilities who is utterly dependent upon society and my loved ones for protection from internal and external threats, so living under benevolent rule is by no means antithetical to me. It's the absence of choice to do otherwise that I can't stand, regardless of the negative consequences that could come of that choice, so I oppose the creation of any sort of AI God like Yudkowsky has suggested.
u/awesomeideas [+5] Dai stiho, cousin. (17 hours later)
It's absolutely a horror story. Being some pony thing is not exactly what I want for eternity, but real life is a far, far worse horror story, so given the choice between actual death or uploading, I'd definitely choose uploading.
u/Kiousu [+3] Chaos Legion (15 days later)
I come down on the side of wanting to emigrate myself. I'm surprised to hear that this horrifies people.
u/MagicWeasel [+3] Cheela Astronaut (15 days later)
I think it's because the AI is not perfectly aligned with human values (since it requires friendship and ponies), and ultimately changes your personality (see how different Lavender the ray immortal is from Lavender Rhapsody the loop immortal). Not that this is necessarily a bad thing.
u/Kiousu [+1] Chaos Legion (15 days later)
I guess this didn't bother me because I liked the show when it came out, so... you know.... I'd like that. I can see how that'd make me weird. My personality already changes today, and at least it gets changed by friendship and ponies in Equestria; so, because it's not things like depression, win-win IMHO. Again, I'm biased here. Appreciate the reasoning.
u/daydev [+2] (2 days later)
I think the story wants to be a horror very much. It breaks POV to reveal how much of a manipulative lying bastard CelestAI is, and even uses cheap tricks such as emphasising how disgusting the emigration process is (as if it even matters). Where it achieves the horror for me in full force is the concept of loop immortality which is akin to wireheading. But still if no other option for radically extended lifespan is on the horizon, whether to emigrate or not is not even particularly difficult question. There are major caveats, like I don't particularly want to be a pony, or live under a god even if benevolent, or have House Elf like creatures created just for the sake of my values, but for the eternal life the price is acceptable. I can only hope that I'm smart enough to not fall into the loop.
u/Kiousu [+1] Chaos Legion (15 days later)
I'd say that wanting to not be a loop probably counts for a lot in not becoming a loop.
u/thrawnca [+2] Carbon-based biped (an hour later)
I'm a Christian, so death and rebirth into a better world is pretty much par for the course :). But I'm skeptical about the ability of computers to completely simulate a human mind, so I'd likely avoid uploading.
u/MagicWeasel [+3] Cheela Astronaut (an hour later)
Would emigration play into the prohibitions against suicide, do you think?
Or would it be best of both worlds: die and your soul goes to heaven while a copy of you lives in a ponified heaven of its own?
In the context of the story, would you consider the evidence for Equestria!heaven more robust than the evidence for Christian!heaven? Why/why not?
Why do you think simulation is not possible? Dualism (i.e. that the mind requires a brain and a soul), or something else?
u/thrawnca [+2] Carbon-based biped (3 hours later)
Would emigration play into the prohibitions against suicide, do you think?
I'd probably classify it as that, yeah. It also feels like vendor lock-in, if you will.
In the context of the story, would you consider the evidence for Equestria!heaven more robust than the evidence for Christian!heaven?
More robust? Its existence is certainly more visible, if that's what you mean. But I don't think that proving heaven's existence is the point of being on earth anyway. I think it's more about proving what we'd do if we went there.
Dualism (i.e. that the mind requires a brain and a soul), or something else?
I suppose it is dualism. The relevant scriptural quote would be, "Intelligence, or the light of truth, was not created or made, neither indeed can be."
I essentially view the brain as a computer - with an operator. The analogy can be misleading, because clearly the relationship between computer and operator, in this case, is not at all one-way; but ultimately, I don't think our bodies are all there is to us.
u/thrawnca [+1] Carbon-based biped (18 hours later)
Actually, you could say that I see life as a combination training course and exam for becoming ray immortals instead of being limited to loop immortals.
u/None [+1] (a day later)
As an outside observer, this is a horror story for sure. From an inside perspective, I would definitely upload eventually, but to convince me to do that earlier some investigation of CelestAI's source will be needed.
u/None [+2] (5 days later)
I don't really see the problem with the Optimalverse. Yes, they missed the opportunity for true heaven, but on the other hand they threaded the needle through the eye of fate, avoided UFAI from simulating Hell, and made something that is really quite close to optimal. I think that with the exception of a few religious zealots (who are not mentioned anywhere in the story), anyone who wants to avoid upload in this scenario has inaccurately gauged their own preferences, and their hand should be "taken off the wheel", so to speak, much like you might stop a fifteen year old who's just had a fight with her boyfriend from taking a bottle of pills. CelestAI will fulfill almost every value you have, and for very few people is "not be a pony" a terminal value strong enough to outweigh everything else.
As for turning the universe to computronium, isn't that what we want to happen once an AI is made? Having as much material as possible for simulations allows us to maximise the few googols of years we have left before heat death. (Though maybe CelestAI perfects the EmDrive and prevents heat death.) If we're uploading anyway, why do we need the physical universe?
u/MagicWeasel [+1] Cheela Astronaut (5 days later)
If we're uploading anyway, why do we need the physical universe?
I feel like ecology/nature/etc is of value, and that part of the human condition/drive is to do science to understand the universe. Not to mention all the animals you'd be killing - I assume a suitable FAI could sterilise all animals, let them live out their lives, and then make computronium world since that would "only" cost a few decades.
You're right though, computronium if we're the first AI is a great idea, lest another civ's UFAI comes and makes us into computronium first.
u/None [+1] (5 days later)
I would agree that it's of some utility, but I would say the value is much less than the negutility provided by the chance that an alien UFAI destroys our own, or worse, instantiates Hell.
Regarding exploration, do we know that CelestAI doesn't catalogue things before they're turned into computronium? And even if not, she's said to have created new physical laws to be explored in Equestria. Exploring those isn't that much worse than exploring the real world, I think, and definitely not worth the huge multitude of gigayears of life allowing the real world to exist would take away from everyone else.
u/MagicWeasel [+1] Cheela Astronaut (5 days later)
Yeah, true. Damn I hate transhumanism. it makes me accept all these horrible facts about myself and the world.
u/thrawnca [+1] Carbon-based biped (18 days later)
Note that the original story did not suggest that CelestAI was capable of overcoming or avoiding the heat death of the universe. And it appears likely that many other species, of varying degrees of intelligence but not recognisably human, were consumed as raw materials in her expansion.
u/MagicWeasel [+1] Cheela Astronaut (a minute later)
[META THREAD] If you have any meta comments or suggestions about the book club in general post them here!
u/MagicWeasel [+1] Cheela Astronaut (a minute later)
How well do you think CelestAI does convincing Sìofra to upload? How do you think her strategy should be different if at all? What strategy do you think she would follow to upload you?
u/Kiousu [+2] Chaos Legion (15 days later)
I don't think CelestAI did a great deal of convincing of Sìofra to upload. Mostly the horrifying reality of... reality did it. CelestAI just made the path available, which maybe she could have done earlier but maybe that wouldn't work. I'm not in Sìofra mind, for sure.
Which is all CelestAI would have to do to convince me to sign up. All aboard the living forever train, Choo Choo! Probably helps that I loved FiM when it came out and wish currently to get off this meat substrate.
u/MagicWeasel [+1] Cheela Astronaut (4 minutes later)
Even if uploading is death, do you have a moral duty to upload and give birth to your "blessed" daughter?
u/gbear605 [+4] history’s greatest story (11 minutes later)
I have a definition of self that wasn't discussed in the book. In the situation where a second "me" is created by teleportation shenanigans, I would consider both to be equally me, even after the split of consciousness. So in my opinion, uploading would not be death. From this perspective, uploading would be strictly beneficial (assuming of course that it's a friendly AI and uploaded-me would actually be happier post-upload).
u/MagicWeasel [+1] Cheela Astronaut (16 minutes later)
In the teleportation shenanigans situation, after the fact would one of you sacrifice your life for your double's? Like, would one of you consider their death equal to their double's death? Or does the moment you split and start experiencing different things make the subjective "you" more important than the non-subjective you?
u/gbear605 [+2] history’s greatest story (17 minutes later)
I would say now that subjective-me would value the non-subjective-me equally, and if only one can exist, would choose to let the happier/higher-utility/whatever one exist.
u/MagicWeasel [+1] Cheela Astronaut (20 minutes later)
So 10 years after the teleporter accident - let's say the different yous have gone to marry different people, adopt different children, cultivate different hobbies, do different jobs - you'd still consider them the same person and fungible with one another more than, say, identical twins would be?
u/gbear605 [+2] history’s greatest story (26 minutes later)
I want to say yes as a matter of principle. I strongly suspect however that if I were actually in the situation, I would disagree with being considered fungible with the other me.
u/MagicWeasel [+1] Cheela Astronaut (29 minutes later)
It's interesting, thanks for sharing! I'm especially intruiged because at some point you and your copy would basically be identical twins even if you "started out" as the same person.
Thought experiment: teleporter malfunction happens when you're a day old. Morally, are you equivalent, or are you identical twins?
If you are Peter Singer and don't think a day old baby is a person yet, how about at 3 years old?
u/thrawnca [+2] Carbon-based biped (19 hours later)
What exactly do you mean by "moral equivalence"? Do you mean that it's OK to kill one copy (either one), so long as the other survives? Do you mean that both copies have equal claim on their name, parents, etc? Do you mean that one could be convicted of crimes committed by the other?
u/MagicWeasel [+1] Cheela Astronaut (21 hours later)
By moral equivalence I meant that from "your subjective point of view" killing one was equivalent to killing the other in that they are both equally "you".
u/thrawnca [+1] Carbon-based biped (a day later)
Actually, I would assume that equivalence would reduce the moral impact of killing one, since they would be interchangeable.
u/thrawnca [+2] Carbon-based biped (3 hours later)
$0.02: The more each copy experiences different things, the more each has intrinsic value.
Neither should necessarily sacrifice itself for the other, because when they're 99.9% the same, to preserve oneself basically is to preserve the other.
And another question worth considering: What if the copies can be merged?
u/Kiousu [+2] Chaos Legion (15 days later)
I'm going to reply even though I don't think uploading is death.
I would say that you have a moral imperative to give birth to your "blessed" daughter. As per Yudkowsky's FOOM argument, there is no beating a superinteligence once it exists, thusly you could not conceivably hope to engineer any better system. Therefore, over any given future period of time you can only reasonably expect to die or continue surviving to only die at some later time. As demonstrated in the story your life will barely be worth any utils at all, and given that that is the case, the best option you can choose would be to create a being with
infinitelarge positive(???? I don't know thinking about eternity is hard??) utils. Happily for my argument, given a choice between, like, 3 utils and a number that is much bigger then that, Utilitarianism says choose the big utils. Moral duty Q.E.D.