## "Malak" - Peter Watts shortfic about a self-modifying AI housed in a combat drone. * Author: u/callmebrotherg *now posting as /u/callmesalticidae** * URL: http://www.rifters.com/real/shorts/PeterWatts_Malak.pdf * Score: 68 * Created: 2016-04-21T00:42:54 ### Post: [Link to content](http://www.rifters.com/real/shorts/PeterWatts_Malak.pdf) ### Comments: > **u/PeridexisErrant** [+21] *put aside fear for courage, and death for life* (2 hours later) > > I think the question here is: *Did the program actually go wrong?* >> **u/EliezerYudkowsky** [+18] *Godric Gryffindor* (20 hours later) >> >> Yes. See, in real life, when you have unintended program behaviors like this, your national phone network crashes, your hard drive gets erased, or your cellphone dials everyone on your contacts list repeatedly at 3am without letting you hear their attempts to call back. Having software this buggy in an armed military drone is likewise strongly net-negative in expectation in real life. A *fictional* story about a case where a software bug just amazingly happens to cause behavior for reasons that a human could map onto moral reasoning, does not mean that it is okay to have heavily armed drones with software of this quality. Good-story bias up the *wazoo.* >> >> I might actually be a bit peeved at this point with the "buggy AI algorithm causes aligned moral reasoning" genre of fiction. Peter Watts executed the trope way better than most, I mean, his AI does not suddenly get inhabited by an anthropomorphic spirit, but this is still NOT WHERE ALIGNED BEHAVIOR COMES FROM IN REAL LIFE. >>> **u/PeridexisErrant** [+9] *put aside fear for courage, and death for life* (23 hours later) >>> >>> This is only broken in the sense that any literal genie is broken though; it carried out it's task of eliminating people who caused civilian deaths. That's not a bug - just a poorly written specification/optimisation goal. And *very poor judgement* from those who deployed it... >>> >>> In short: the program is a tool, not an agent, and it's behaviour is merely unexpected (not unforeseeable or unpreventable). No moral reasoning takes place in the machine. All it does is resolve the conflict between human-supplied 'ethical architecture' and the action of humans. >>>> **u/EliezerYudkowsky** [+16] *Godric Gryffindor* (a day later) >>>> >>>> And by an *amazing coincidence* this unexpected behavior corresponds to a fine morality tale, rather than firing on children wearing straw hats whenever the moon is half-full and it's dropped a prime number of munitions in the last 37 minutes. >>>>> **u/callmebrotherg** [+6] *now posting as /u/callmesalticidae* (a day later) >>>>> >>>>> That's the sequel. >:P >>>>> **u/AugSphere** [+13] *Dark Lord of Corruption* (a day later) >>>>> >>>>> I understand your frustration, but come on. Not all unexpected behaviours are equally unexpected. Bugs are not the result of some unknowable eldritch influence. Given the amount of testing such a system would presumably go through, it's likely that the resulting AI behaves *almost* as intended, rather than displaying some seemingly random high-complexity behaviour. >>>>> >>>>> I think there is even a case to be made that the AI is actually functioning properly as the people who programmed it intended. Saying that the AI is malfunctioning in this case is alike to declaring a microwave oven to be poorly designed because some moron decides to stick a living cat into it and turn it on. Engineers cannot foresee and prevent every way the idiots might misuse their product. They intended to install an 'ethical architecture' and they did that, it's not their fault that humans don't actually behave ethically (as defined by the said reasonable architecture). I'm mostly with /u/PeridexisErrant on this. >>>>> >>>>> I guess it comes down to the question of whether the humans overriding the ethical constraints were actually acting reasonably or not. My impression from reading the story was that they were acting unethically, in which case the AI's ethical architecture recognising that fact doesn't really count as a failure. >>>>> >>>>> Although, given the presumed difficulty of alignment problem, I guess one can trust this AI to be faulty in some way just on priors. From this perspective, it's surprising that it's faulty in this particular superficially moral way. >>>>>> **u/Burns_Cacti** [+1] (4 months later) >>>>>> >>>>>> 5 months late but, I assumed they were acting ethically in a *greater good* sort of way, but the ethical architecture has no frame of reference for those sorts of calls. It can only judge what's immediately in front of it. >>>>> **u/PeridexisErrant** [+6] *put aside fear for courage, and death for life* (2 days later) >>>>> >>>>> > And by an amazing coincidence this unexpected behavior corresponds to a fine morality tale >>>>> >>>>> That's not a coincidence, that's the **ethical architecture** that the UN insisted was built in. Isn't your major work to ensure humans can determine the moral framework of machine intelligence? Here's an example! >>>>>> **u/EliezerYudkowsky** [+4] *Godric Gryffindor* (2 days later) >>>>>> >>>>>> I can't tell whether you're trying to defend the story on a literary level or claim that this is the sort of thing that would happen with reasonable probability in real life. On a literary level, Peter Watts did an excellent job of rationalizing something that wouldn't happen. On a real life level, your strength as a rationalist is to be more confused by rationalizations of things that wouldn't happen in real life than things that would, so you need to see past literary devices and say 'but nonetheless, in real life...' as part of that skill. >>>>>>> **u/PeridexisErrant** [+10] *put aside fear for courage, and death for life* (2 days later) >>>>>>> >>>>>>> My original contention was that, arguably, the drone was carrying out the intention of it's designers - who did not anticipate it's operators crimes. >>>>>>> >>>>>>> My second point was that the drone *did not* spontaneously derive moral principles - they were explicitly built in (cf 'ethical architecture'). >>>>>>> >>>>>>> Yes, this would be staggeringly unlikely to happen in real life. I'm not claiming otherwise - simply that the events depicted don't involve unintended target selection (merely unanticipated), nor moral behaviour emerging from any source but deliberate input. >>>>>>>> **u/tbroch** [+3] (4 days later) >>>>>>>> >>>>>>>> It's usually very hard to properly predict how software will function without a lot of testing and debugging, even for very simple tasks. For something like a complex morality function, it's seems very, very unlikely to me that it would work in unexpected ways that happen to line up with the designers desires. In the real world, for every unplanned accident that leads to a surprisingly good result, there are 99 accidents that just lead to failure. >>>>>>>> >>>>>>>> The fact that the creators of a complex function did not rigorously plan for and expect a certain happy outcome is itself evidence that the happy outcome should not be expected. >>>>>>>> **u/EliezerYudkowsky** [+4] *Godric Gryffindor* (2 days later) >>>>>>>> >>>>>>>> No Just No. The only reason that architecture led to a moralish result was because The Author Said So. That quality of programming would not pinpoint that result in real life, and would instead slaughter children on the full moon, so you cannot say that morality was built into the stated architecture or emerged from it. Peter Watts can make it happen in the story; he can't make it a natural or logical consequence. >>>>>>>>> **u/Ristridin1** [+2] (3 days later) >>>>>>>>> >>>>>>>>> I have to ask. Why is it more logical that the ethical architecture results in the slaughter of children on the full moon than the actual end of the story? >>>>>>>>> >>>>>>>>> Obviously, a lot of the programming is very implicit. Most importantly, one major unanswered question is how exactly the program assigns colors to targets. This seems like a major challenge, and would probably be something that could cause a slaughter of children on a full moon (or any of billions upon billions of unintended consequences) by marking them as reds for some reason. I would say however that the in-story assumption is that that part of the programming has been done. Aside from that, some ethical architecture was 'grudgingly put in'; this is the part that results in the unintended behavior (and has possibly not been tested carefully, since the army probably didn't care about that too much). >>>>>>>>> >>>>>>>>> [Spoiler](/s "IF expected collateral exceeds expected >>>>>>>>> payoff THEN abort UNLESS overridden. >>>>>>>>> IF X attacks Azrael THEN X is Red. >>>>>>>>> IF X attacks six or more Blues THEN X is Red") >>>>>>>>> >>>>>>>>> "Children on the full moon" do not violate the above rules; they don't 'attack', 'cause attack' or 'override abort'. Mission control does. >>>>>>>>> >>>>>>>>> I can definitely see other outcomes than the one in the story, but nothing like the 'children on a full moon' scenario. For example, Azrael could self-destruct, since in principle, Azrael attacked six or more blues. This at least seems to require fewer logical steps than mixing up the three definitions above. >>>>>>>>> >>>>>>>>> So why the slaughter of children? Do you have some set of ethical rules in mind that would cause the slaughter of children in fewer steps than those outlined in the story? >>>>> **u/None** [+2] (2 days later) >>>>> >>>>> That story would probably not be as interesting. >>>>> **u/TangleF23** [+1] (6 years later) >>>>> >>>>> Eliezer Yudkowsky, everyone: Pissed that a story designed to have a moral point has a moral point. > **u/EliezerYudkowsky** [+26] *Godric Gryffindor* (an hour later) > > Not self-modifying! The whole point of the story is that these were just fixed update rules doing their thing. > > Can someone fix title? >> **u/callmebrotherg** [+5] *now posting as /u/callmesalticidae* (an hour later) >> >> True. I was trying to be concise, but I admit that it gives the wrong impression. >> >> Sorry! :< >> **u/PeridexisErrant** [+1] *put aside fear for courage, and death for life* (2 hours later) >> >> Added post flair, hopefully close enough. >>> **u/TennisMaster2** [+1] (2 hours later) >>> >>> Can you add, "not self-modifying"? >>>> **u/PeridexisErrant** [+1] *put aside fear for courage, and death for life* (3 hours later) >>>> >>>> Yes! Of all the things I tried that were too long, I didn't test the obvious... >>>>> **u/EliezerYudkowsky** [+6] *Godric Gryffindor* (20 hours later) >>>>> >>>>> It *is* modifying. It's not *self*-modifying. A neural network updates, but it doesn't write better neural network algorithms using consequentialism. >>>>>> **u/PeridexisErrant** [+2] *put aside fear for courage, and death for life* (23 hours later) >>>>>> >>>>>> I know, but you can't fit `self-` and `PDF` in the flair length limit. :( > **u/TheJamesRocket** [+2] (a day later) > > A fantastic story. Peter watts handles AGI in a far more realistic manner than most authors. >> **u/None** [+4] (a day later) >> >> It's not a general intelligence, though... it's a military drone that does military drone-y things, plus a bug that changes its behaviour slightly to break horribly. >>> **u/TheJamesRocket** [+2] (a day later) >>> >>> Actually, you have a point on that. Malaks 'thought process' seems compliant with the standard narrow AI paradigm. >>>> **u/Corticotropin** [+1] (18 days later) >>>> >>>> Seems like a neural network, fed the results of an exceptional machine vision algorithm, which has the reward function of cost/benefit (and previous was just benefit). >>>> >>>> The part I believe less is the absurdly good machine vision algorithm ;P > **u/Dwood15** [+1] (an hour later) > > Very interesting and well-written short story! > **u/xamueljones** [+1] *My arch-enemy is entropy* (2 hours later) > > Just warning people who read comments before clicking on the story that this is a download link, not a link to a story posted online. > > Very good story relating AI to war. Worth the read and I don't want to say anything more, because it's so short that I could spoil something just by talking about it. >> **u/None** [+4] (6 hours later) >> >> [deleted] >>> **u/para-di-siac** [+2] (18 hours later) >>> >>> On the phone, the in-app browser opens the link as a downloadable file >>>> **u/343427229486267** [+1] (5 years later) >>>> >>>> On your phone, on the browser you used, that one time... >> **u/callmebrotherg** [+4] *now posting as /u/callmesalticidae* (2 hours later) >> >> Hm. Maybe it's a Chrome thing, but when I click on the link it appears in my browser without downloading. >> >> I'm sorry if something different happens with other browsers (or if I changed a setting a long time ago and forgot), and I'll keep that in mind for the future. >>> **u/None** [+4] (a day later) >>> >>> Technically, if it appears in your browser, it is downloaded (just not saved to persistent storage). >>> **u/None** [+2] (2 days later) >>> >>> I think it's a phone thing. >>> **u/GaBeRockKing** [+0] *Horizon Breach: http://archiveofourown.org/works/6785857* (16 hours later) >>> >>> chrome opens PDFs as new tabs, while firefox just downloads them. >>> **u/earfluff** [+1] *Ankh-Morpork City Watch* (4 hours later) >>> >>> It appears to me without downloading. > **u/BassoeG** [+1] (7 years later) > > [Once again, Watts was a prophet.](https://www.vice.com/en/article/4a33gj/ai-controlled-drone-goes-rogue-kills-human-operator-in-usaf-simulated-test) >> **u/callmesalticidae** [+1] *writes worldbuilding books* (7 years later) >> >> Some folks get annoyed by comments on old posts, so I want to explicitly thank you. This is a really interesting article and I endorse the resurrection of dead threads. ---