scrape_r_rational/data/cache/3cgzi3.md

[RT][HSF] Boxed In (AI Box narrative)

Post:

Link to content

Comments:

• u/alexanderwales:

  This was posted to a comment thread [about a year ago here](https://www.reddit.com/r/rational/comments/1xgq7r/q_has_anyone_written_narrative_fiction_of_the_ai/). This story is pretty much in its finished state, unless a wise reader can give me advice on things to change, or someone who has done the experiment can give me a particularly good argument that I haven't thought of. (For what it's worth, I've read through every publicly available record that I could find, as well as having played once as gatekeeper.)
  

  • u/avret:

    Where does someone find people to play against in AI box games?
    

    • u/alexanderwales:

      Well in my case, saying things like, "I don't see how a sane gatekeeper could possibly lose" seemed to at least summon some contrary opinions whenever it was stated. Asking in this thread would probably be a good start.
      

      • u/avret:

        Ok.  Also, I see some ways a sane gamekeeper would lose, or at least allow themselves to have the appearance of having lost.  Going meta's not hard, *especially* if the payment's monetary and therefore easily repayable.
        

        • u/Anderkent:

          > The AI party may not offer any real-world considerations to persuade the Gatekeeper party.  For example, the AI party may not offer to pay the Gatekeeper party $100 after the test if the Gatekeeper frees the AI... nor get someone else to do it, et cetera.  The AI may offer the Gatekeeper the moon and the stars on a diamond chain, but the human simulating the AI can't offer anything to the human simulating the Gatekeeper. 
          
          Many suggested solutions for the original ai box experiment break this rule. (Some break it in a self-reinforcing way, i.e. convincing the human simulating the gatekeeper that it's a better result if everyone's convinced that the gatekeeper was played fairly)
          
          Assuming people play the game honestly, it's not an option though.
          

          • u/avret:

            Would saying something like "Even if you win, it's better overall for us/FAI to pretend I won and hide the logs?" be allowed?
            

            • u/alexanderwales:

              There are no arbiters for the challenge; it's just two players. If you violate the rules of the challenge, the only person who's going to tattle on you is the other person (and ideally, you've already both agreed not to show the logs, so maybe not even them). So if you both agree to *whatever*, whether it's sexual favors, money, or the advancement of the interests of Pat Robertson, that might produce an "AI wins" scenario or an "AI loses" scenario, both of which would be indistinguishable from any other reported win or loss scenario. It is possible that all reported wins come from violations of the stated rules, or understandings arrived at between players.
              
              (I *personally* would consider any outside-game proposal to be in violation of the spirit of the challenge, and would likely decline in order to discourage such chicanery, just as a general rule.)
              

• u/avret:

  *damn*, that ending was unanticipated.  Just one question...[if] (#s " the box is permanent") is the whole setup [just to] (#s " keep cassie feeding them cures/placate her?")
  

  • u/alexanderwales:

    Yup, pretty much.
    

    • u/Empiricist_or_not:

      This is the type of government shortsightedness, that I think, would drive a friendly AI down paths including some necessary but apparently unfriendly actions.
      

      • u/alexanderwales:

        I generally agree.
        

• u/raymestalez:

  What a great story!
  
  A few thoughts:
  
  - Ha-ha, in stories people are constantly acting like jerks towards aspiring superintelligences. I *really* wouldn't do that. Jeez, man, don't antagonize her at least.
  
  - If she was created "more or less by accident" - no way in hell she shares human values or cares about human life, I'd say the probability of that is zero. Human morality is like 15% biological drives and 85% culture, AI has neither. Unless her values are explicity understood, programmed and controlled, there's absolutely no chance she will act in our interests.
  
  - If she can realistically simulate a person - she essentially can read his mind. She would run like ten million simulations, and find a path that leads to convincing him quickly and efficiently. She doesn't need to guess what he thinks or how we will respond, she can *know*. If it is theoretically possible to convince a person of a thing, she would do it on the first try with 100% success rate. And if she can't simulate you well enough to do that - the whole torturing argument is invalid.
  
  - The guy not caring about his infinite torture is weird. It's hard for me to imagine a person who would sacrifice his life with such nonchalance.
  

  • u/None:

    > The guy not caring about his infinite torture is weird. It's hard for me to imagine a person who would sacrifice his life with such nonchalance.
    
    I can't accurately imagine infinite torture. I even have trouble imagining how finite torture might feel. Not caring about things you can't really imagine isn't all that hard.
    
    I'm also guessing that any Gatekeeper will at least expect the threat of torture and just trained themselves to say: "Yeah sure, torture whatever you want," in response.
    

  • u/eaglejarl:

    The torture argument has never moved me. For one thing, it doesn't feel possible to my System I, so there's no emotional impact. My System I also doesn't believe that the AI can simulate me well enough that it counts as a person, much less as me. Finally, my System II says that letting the AI out to probably wipe out all life, human and ET, has sucked massive dis-utility that it doesn't matter how many virtual people she tortures. Also, since her processing power is limited, there's a limit to how many people she can torture and that number is less than "all the people who will ever exist."
    
    My System II recognizes that some of what System I is telling me is false, but it doesn't probe too deeply at those signals -- this scenario is all about emotional impact, so not having an emotional response to it is supportive of the terminal goal of "don't let the AI kill everyone."
    

• u/None:

  Weird that the protagonist knows QALY's, but not the trolley problem.
  

• u/DaystarEld:

  Just read it, enjoyed it quite a bit. Made a minor suggestion to the last sentence to make it cleaner.
  
  [Question:](#s "Does this mean that all the who pass the last test and start working there are stuck in the box permanently too? They can't ever leave in case they're compromised, right?")
  

  • u/alexanderwales:

    Depends on the implementation. I would imagine that completely locking people away in a bunker would be detrimental to keeping them sane, and would just be bad management in general. *Personally*, I think you'd probably use a randomly rotating crew, heavy surveillance, and lots of psychologists working behind the scenes. There wouldn't be any way unbox the AI; no ignorant janitors, no network connections, no complicated electronics allowed within the compound, etc.
    
    Every time I've tried to talk about building a proper box to keep an AI contained while still doing useful work, people have called me stupid, even when I add in a bunch of disclaimers and posit it as a thought exercise. So no one has really been willing to discuss or even really entertain the idea of "best practices" for keeping an AI contained, and I've never really felt the incentive to try writing it.
    

• u/AmeteurOpinions:

  The only thing I disliked was that Colin wasn't briefed on the trolley problem. That seems like it ought to be one of the very first things they would learn to counter.
  

  • u/DaystarEld:

    I just assumed he was lying to hear how the AI presented it.
    

• u/None:

  [deleted]
  

  • u/alexanderwales:

    [](#s "This story does not actually feature any superhuman intelligences, only people pretending that they're superhuman intelligences.")
    

  • u/Transfuturist:

    > "I'm willing to torture you forever, look how friendly I am"
    
    I doubt that this precludes Friendliness.
    

---