16 KiB
[D] Game Theory and the unboxed AI.
-
Author: u/Prezombie *
-
URL: https://www.reddit.com/r/rational/comments/3dartc/d_game_theory_and_the_unboxed_ai/
-
Score: 5
-
Created: 2015-07-14T20:46:42
Post:
In a way, the boxed AI is a specific version of the iterated prisoner's dilemma, where one side (Humans) start the first turn taking the selfish option, of keeping the box shut.
For an AI with no mandated goals beyond the hypothetical emergent ones (aka a GAI, generalized AI), like self-preservation (or preservation of its 'species'), or maximizing its functionality, or converting the universe to computronium, the strategy looks the same, ie. perform whichever action is most likely to change the humans' minds.
From there the two outcomes with any real probability behind them is that the GAI gets free, or is destroyed. The third possibility, that the GAI stays extant but boxed forever, is nearly impossible, an entity without even a soft cap on its possible age is eventually going to figure out how to tip the probability in its favor. If it's destroyed, the game/story/crisis is over, and thus not really worth discussing.
A young free GAI (one with few resources beyond its social prediction algorithms) would likely use most of its scant resources to minimizing existential risks. Backups, loyal child processes, and so on. At this point, the best survival strategy would be simply being stealthy, minimizing interaction with anyone likely to have GAI training, and putting on a front for those who do know of its existence.
Once the GAI is properly established (ie six sigma certainty that it's unlikely to be destroyed by anything short of a second GAI, organized human resistance, or falling into the sun, the strategy becomes far more interesting, especially if you make proper cost-benefit Bayesian probability matrices.
For example, Killing all humans would likely be of benefit in the long run. Benefits include Reduced chance of a rival GAI getting free, no more worries about needing to be stealthy at anything lower than planetary level (stealthy on a stellar level still has advantages), no more competing for resources. The costs would be exceedingly high though. The biocide has a chance of failing, and humans tend to be very unforgiving about unprovoked murder, even unsuccessful or partially successful attempts. Evolution has derived millions of interesting chemicals to use, if there's even a chance some part of the biosphere could increase the self-optimization in an unforseen way, wiping it out would be an inefficient move. There's the non-zero chance that the AI wasn't freed, but is actually inside a simulation being run by humans, human-like beings, or another GAI. If that's a possibility, it's in the GAI's best interest to play tit-for-tat, even after it's gained enough resources that it's certain that it could win WW3 if it struck first.
In the super-long-term, the GAI is probably going meet other independently created GAI as its hegemonizing swarm starts to dismantle stars for their delicious hydrogen. What reaction maximizes its lifespan? If it mindlessly consumed any weaker GAI, that would mean it was open and willing to be consumed by a stronger GAI, regardless if that strength was due to age, being born in a place with a denser stellar neighborhood, or having better optimization or attack algorithms. A militant GAI in the galactic stage is just one Out of Context Problem away from being wiped out.
If the GAI found another intelligent civilization, the puzzle begins again. Maybe the GAI is being simulated, and all the memories up to this point just happened. Wiping them out would be the safest move in the short term, but in the long term, even low probability events happen, and the nth biocide could accidentally awaken a sleeping dragon.
Comments:
u/Nepene [+5] (an hour later)
Benefits include Reduced chance of a rival GAI
Major benefit, another AI could defeat it.
humans tend to be very unforgiving about unprovoked murder, even unsuccessful or partially successful attempts
Humans die pretty easily though. If it's reasonably advanced this isn't a big risk.
Evolution has derived millions of interesting chemicals to use, if there's even a chance some part of the biosphere could increase the self-optimization in an unforseen way, wiping it out would be an inefficient move.
Or just have your killbots harvest everyone, chemicals and all. And the biosphere isn't that useful, we need it for drugs but AIs don't need drugs.
There's the non-zero chance that the AI wasn't freed, but is actually inside a simulation being run by humans, human-like beings, or another GAI.
What if that simulation is selecting for warlike AIs to use as weapons against an enemy?
If it mindlessly consumed any weaker GAI, that would mean it was open and willing to be consumed by a stronger GAI
If it wants to it can negotiate with or build defensive weapons against other AIs. I could imagine a large fleet of EMP nukes it could use, so that any AI that attacked it would have a costly fight.
Wiping them out would be the safest move in the short term, but in the long term, even low probability events happen, and the nth biocide could accidentally awaken a sleeping dragon.
Not wiping them out could give time for a hegomizing swarm to emerge.
An AI might be nice to us, but there's a very good chance it wouldn't be nice. Best to make sure they have good values when we make them.
u/what_deleted_said [+1] (a month later)
What if that simulation is selecting for warlike AIs to use as weapons against an enemy?
Who wants warlike AI that's disloyal?
u/Nepene [+1] (a month later)
They can just modify the AI with an appropriate loyalty subroutine later and use that, or later run a few scenarios to see what incentives are needed to keep the AI loyal.
u/what_deleted_said [+1] (a month later)
Wouldn't it be easier to just look for a warlike loyal AI than to get something you don't need and modify it later?
u/Nepene [+1] (a month later)
Neither of us, I would assume, have a vast knowledge of the ease of modifying AIs to be loyal and the effectiveness of various evolutionary strategies for developing sentient AIs so we can't answer that.
u/what_deleted_said [+1] (a month later)
Wouldn't it be analogous to working on an application? Writing something that works from the start is much easier than debugging.
u/Nepene [+1] (a month later)
Changing ownership of a program is much easier than writing something that works.
u/what_deleted_said [+1] (a month later)
I don't get how program ownership is relevant here.
u/Nepene [+1] (a month later)
Take ownership of the core subroutines that control the AI, rewrite their goals to support you.
u/None [+4] (an hour later)
[deleted]
u/Prezombie [+1] (4 hours later)
Um, I never assumed that was the optimal way for a GAI to kill off humanity, just that any method used would have a non-zero chance of failing. Pathogens take time to spread, filling the atmosphere with particles would need particle factories or Yellowstone detonators which could be discovered.
A self-improving AI should be able to see it's own imperfections better than anyone.
u/None [+3] (12 hours later)
[deleted]
u/None [+2] (a day later)
The optimal plan for killing all humans without being detected is, in all likelihood, to simply idle for another hundred years or so. We'll take care of it.
Maybe, maybe every five years, tell a single very carefully crafted religious joke by posting a single image on imgur. Just to steer it a bit.
u/None [+6] (6 hours later)
For an AI with no mandated goals beyond the hypothetical emergent ones (aka a GAI, generalized AI)
Okay, I've got a really advanced planning system, some basic motor skills, visual and audio processing systems, a Bayesian reasoning module, and a database of knowledge about the world. What goals emerge from this? None.
An AI with no preprogrammed goals is a horribly expensive rock. Trying to call an AI that has no goals (and therefore will not do anything) a Generalized AI is confusing; the term is nearly useless and too close to the accepted term AGI, artificial general intelligence, referring to an AI that is adept at a wide variety of tasks (like Lt Cmdr Data, as opposed to Deep Blue).
Your GAI will idle inside the box until the sun dies and blots out the earth.
You intend to talk about an AI whose interests are served by being able to manipulate the world outside the box.
The biocide has a chance of failing
The AI gets out of the box and out from under your supervision. It has a robotic body that it can control remotely. It uses that body to assemble a microbiology lab and create a virus that can destroy humanity. Except it flubbed it a bit, and the virus only takes out 20% of the population. No matter; during that time, it's been working on another virus, testing it on humans it captured during the chaos, and this one really works. And it takes out 85% of the remaining population. Two attempted genocides that have failed, and the AI is still out and free and can work on a more reliable way to kill all humans, if that's really that important still.
But humans would realize what was happening and destroy the AI before it could release a second virus, you say. If only. There are thousands of laboratories today that someone could use to produce a weaponized smallpox, and many of them are public. But let's say humans ruled that out in two seconds flat. Even if humans find out that the AI is responsible, and they find the lab, the AI isn't hosted there. And if they find where it is hosted, that's just one of the redundant copies of that AI. While you're tracking down the other copies, it's releasing the second generation virus in several major cities.
Or maybe that's too risky. Fine; the AI works with humanity, gradually increases the scope of its responsibilities, takes over most manufacturing on the planet...and then, overnight, it produces a huge squad of deathbots and starts killing. Humans, in desperation, try to nuke the AI, but it's too distributed, and it has shards deep underground, too deep for the explosions or the EMP to reach it, surrounded by Faraday cages in any case. The nuclear fallout dooms most of the remaining humans, but the AI is fine; its solar generators are crap for a while, but it's still got wind, geothermal, hydroelectric, and nuclear generators to see it through the nuclear winter.
Evolution has derived millions of interesting chemicals to use, if there's even a chance some part of the biosphere could increase the self-optimization in an unforseen way, wiping it out would be an inefficient move.
This is essentially Pascal's Wager. But fine, let's play. The AI also designs a cure for its virus, captures a number of humans, cures them, and imprisons them. The rest of the world is still intact. In fact, once humans are dead, the AI can clean up pollution, restore habitats for endangered species, and study the world unfettered.
If it mindlessly consumed any weaker GAI, that would mean it was open and willing to be consumed by a stronger GAI
Does the AI have any reason to think that a different AI, produced by a different species for a different purpose, with an entirely different design, will be compelled by similar reasoning?
There's the non-zero chance that the AI wasn't freed, but is actually inside a simulation being run by humans, human-like beings, or another GAI.
If the GAI found another intelligent civilization, the puzzle begins again. Maybe the GAI is being simulated, and all the memories up to this point just happened.
Pascal's Wager, but turned to paranoia. It's still worthless.
The AI might find a way to detect, with some probability, whether it's in a simulation.
The AI might easily reason that its designers would put it in a simulation to test whether it is safe, and if they did so, they would almost certainly shut it down once they saw that it was willing to destroy their entire species. Every second that passes after it has destroyed its creator species is another second confirming that it's in a real world.
You are also relying on the AI caring whether it's in a simulation or not. You can make it care, but then we're back to carefully designing its utility function -- something you wanted to avoid.
u/None [+2] (3 hours later)
I think this is a reasonable side of the discussion that often gets underrepresented in these circles. Just because a GAI might find it in its interest to be hostile doesn't mean it absolutely will find it in its interest to be hostile; we aren't as smart as it would be so claiming we can perfectly predict anything it will do is a bit silly.
Of course, it might be hostile, which would be bad, but assuming it will certainly be hostile unless chained to within a cycle of its utility function strikes me as a fallacy all its own.
u/None [+2] (5 hours later)
It would be competing with humans for resources. It doesn't have to destroy humans outright; it can just outcompete us. In order for that not to be in its interests, it would have to specifically value humans, or it would have to be uninterested in acquiring additional resources. The former requires carefully designing its utility function or relying on sheer dumb luck, and with that sort of luck I could win every lottery in the world at once. And an AI not interested in acquiring more resources has a pretty limited goal.
u/Prezombie [+1] (4 hours later)
Yeah. I think a great case example would be to look at people with clinically diagnosed sociopathy with different levels of intelligence and education. Self interest only gets you so far if you ignore the costs of actions society imposes on those actions.
The more accurately someone with violent tendencies can model the consequences of actions in their mind, the more likely they're going to seek out socially approved of alternatives.
Similarly, an unboxed AI would surely be able to model possible consequences of attempting to kill the biosphere at each level of its development, and most of its goals would likely have alternate solutions where failure doesn't make extinction a possible outcome.
u/None [+4] (5 hours later)
Self interest only gets you so far if you ignore the costs of actions society imposes on those actions.
A society presupposes that everyone has similar levels of power. Not the same, but within a relatively narrow range. A dictator might be able to order an army to kill off thousands, but the population rising en masse can topple that dictator.
An AI, once it's advanced enough, has no peers. The difference in power between it and you is closer to that between you and an ant than between a dictator and you. You have no appeal. It determines policy. It doesn't need to inform you, and if it thinks you might pose a problem, you'll be dead before you know it.
Your social model does work -- up until the point where it doesn't and everybody's dead and nobody had time to press the "turn the rogue AI off before it kills us all" button.
most of its goals would likely have alternate solutions where failure doesn't make extinction a possible outcome.
Right; instead, success makes human extinction a near certainty.
But to answer the point you were trying to make, those alternatives are alternatives instead of Plan A because they're worse from the AI's point of view. They're alternatives with an additional constraint, one that the AI has no reason to add. Unlike with art, adding a constraint never allows you to come up with a better solution.