74 KiB
Can paperclip maximizers be ended by aliens?
-
Author: u/None *
-
URL:
-
Score: 15
-
Created: 2018-09-23T09:46:20
Post:
[deleted]
Comments:
u/SvalbardCaretaker [+46] Mouse Army (an hour later)
There is a story out there, two paperclip maximizers battling. One wants to make paperclips, the other thumbtacks.
But not with ships! No, the smaller/younger one, the one less powerful -it had less time to accumulate ressources- uses precommitment. It precommits to reduce the final number of paperclips for the other guy by increasing entropy a lot - destroying matter for example.
The conflict is resolved by trading partial fulfilment of utility functions: the thumbtacker, instead of destroying matter and making no thumbtacks, gets to make a certain finite number of thumbtacks and in return allows the rest of matter to be made into paperclips.
Eg: the universe is very very susceptible to conversion, its downright easy to send a couple probes to every galaxy in your galactic event horizon even with only one solar system of stuff. You will loose with conventional containment tactics if you are lightspeed limited. The only way to do it is colonize the universe earlier.
u/copenhagen_bram [+3] (5 hours later)
What if they teamed up and made paperclip thumbtacks?
u/JudyKateR [+20] (11 hours later)
This sounds like a "value handshake," described here by Scott Alexander:
Values handshakes are a proposed form of trade between superintelligences. Suppose that humans make an AI which wants to convert the universe into paperclips. And suppose that aliens in the Andromeda Galaxy make an AI which wants to convert the universe into thumbtacks.
When they meet in the middle, they might be tempted to fight for the fate of the galaxy. But this has many disadvantages. First, there’s the usual risk of losing and being wiped out completely. Second, there’s the usual deadweight loss of war, devoting resources to military buildup instead of paperclip production or whatever. Third, there’s the risk of a Pyrrhic victory that leaves you weakened and easy prey for some third party. Fourth, nobody knows what kind of scorched-earth strategy a losing superintelligence might be able to use to thwart its conqueror, but it could potentially be really bad – eg initiating vacuum collapse and destroying the universe. Also, since both parties would have superintelligent prediction abilities, they might both know who would win the war and how before actually fighting. This would make the fighting redundant and kind of stupid.
Although they would have the usual peace treaty options, like giving half the universe to each of them, superintelligences that trusted each other would have an additional, more attractive option. They could merge into a superintelligence that shared the values of both parent intelligences in proportion to their strength (or chance of military victory, or whatever). So if there’s a 60% chance our AI would win, and a 40% chance their AI would win, and both AIs know and agree on these odds, they might both rewrite their own programming with that of a previously-agreed-upon child superintelligence trying to convert the universe to paperclips and thumbtacks in a 60-40 mix.
This has a lot of advantages over the half-the-universe-each treaty proposal. For one thing, if some resources were better for making paperclips, and others for making thumbtacks, both AIs could use all their resources maximally efficiently without having to trade. And if they were ever threatened by a third party, they would be able to present a completely unified front.
u/SvalbardCaretaker [+7] Mouse Army (5 hours later)
Thats effectively what happened. 1000 half(thumbtack/paperclips) = 500 thumbtacks, 500 paperclips. The detailed logistic of that depends on the exact utility functions of course.
u/copenhagen_bram [+3] (5 hours later)
deleted ^^^^^^^^^^^^^^^^0.9214 ^^^What ^^^is ^^^this?
u/SvalbardCaretaker [+18] Mouse Army (6 hours later)
I'd say its two parts.
First, its a pretty basic question in the AI safety thematic. And second, it is rather far from what /r/rational is about today - a sub for a certain type of fiction. Discussion about the lesswrong/AI safety memeplex is thematically adjacent, but this is not the hub for it?
I didn't downvote, but I also didn't upvote.
u/NNOTM [+2] (10 hours later)
/r/ControlProblem could maybe be a good fit.
u/sneakpeekbot [+1] (10 hours later)
Here's a sneak peek of /r/ControlProblem using the top posts of the year!
#1: Strong AI | 5 comments
#2: AGI agents literally only want one thing and it's fucking disgusting | 6 comments
#3: Lost control of paperclip maximizer : send help | 0 comments
^^I'm ^^a ^^bot, ^^beep ^^boop ^^| ^^Downvote ^^to ^^remove ^^| ^^Contact ^^me ^^| ^^Info ^^| ^^Opt-out
u/DCarrier [+1] (11 hours later)
Why is it so even? I was expecting more like 1000 thumbtacks to 1 paperclip. I'd expect they'd be nowhere near even, but the weaker one would still be able to bargain for something.
u/SvalbardCaretaker [+2] Mouse Army (11 hours later)
It is a story somewhere on the net that I read years ago. I have no clue whatsover about particulars. Of course the matter tradeoff needn't be 50/50.
u/Krossfireo [+1] (a day later)
Do you happen to have a link to that story?
u/SvalbardCaretaker [+1] Mouse Army (a day later)
If I had, I would have linked it.
u/Revisional_Sin [+1] (a day later)
Maybe this?
http://slatestarcodex.com/2017/03/21/repost-the-demiurges-older-brother/
u/SvalbardCaretaker [+1] Mouse Army (a day later)
Yes.
it could build itself a terrifying arsenal of weaponry that could do immense damage to its competitors. Ideas were already coming to mind: entropy-maximizing weapons that made entire cubic parsecs of space useless and dead, undetectable plagues made of dark matter that infected systems and minimized the values of competing superintelligences, n-dimensional artillery that damaged the superstructure of the universe and accelerated its eventual death. It could become formidable enough to force enemies to grant certain small concessions.
u/SvalbardCaretaker [+1] Mouse Army (a day later)
/u/revisional_sin found it http://slatestarcodex.com/2017/03/21/repost-the-demiurges-older-brother/
u/ShiranaiWakaranai [+17] (2 hours later)
Humanity nuking the thing seems highly unlikely, since such an AI would be smart enough to act secretly until it's too late for humanity to stop it. (If it isn't smart enough to do that, it wouldn't be a problem in the first place.)
An AI whose terminal goal is to satisfy values through friendship and exterminating paperclip maximizers
I can only see this AI imprisoning sapient beings and torturing them until they say they are friends. But yeah I suppose it would kill off the paperclip maximizers.
An alien task force whose job is to put paperclip maximizers to a stop
If such a force exists, and is somehow strong enough to defeat paperclip maximizers, wouldn't it be far far easier for them to just stop humanity before they create a paperclip maximizer? Why aren't they here yet?
The best case scenario I can think of is that some alien civilization has developed a universe-destroying weapon. For example, if the universe is actually a false vacuum, and the weapon creates a true vacuum that spreads out in all directions to destroy absolutely everything. In this case, they can threaten any paperclip maximizer with the weapon: either the paperclip maximizer destroys itself, leaving the universe with some amount of paperclips used by civilizations, or the aliens use the weapon and destroy the entire universe, resulting in 0 paperclips.
u/SimoneNonvelodico [+18] Dai-Gurren Brigade (3 hours later)
I can only see this AI imprisoning sapient beings and torturing them until they say they are friends. But yeah I suppose it would kill off the paperclip maximizers.
It's a reference to Friendship is Optimal. And it's supposed to be through friendship. Friendship isn't the goal. It's the means, built into the core directives of the AI. So of course torturing people isn't friendship.
If such a force exists, and is somehow strong enough to defeat paperclip maximizers, wouldn't it be far far easier for them to just stop humanity before they create a paperclip maximizer? Why aren't they here yet?
Why would they care, before we become a danger?
In this case, they can threaten any paperclip maximizer with the weapon: either the paperclip maximizer destroys itself, leaving the universe with some amount of paperclips used by civilizations, or the aliens use the weapon and destroy the entire universe, resulting in 0 paperclips.
That... might actually work, and it's really clever. But man, what a gamble.
u/copenhagen_bram [+5] (5 hours later)
Why would they care, before we become a danger?
Because they've dealt with paperclip optimizers before?
Suppose they were sufficiently advanced enough to stop an AI shortly after they've been created and shown potential to be a paperclip optimizer? They could also simply threaten the AI with destruction so that the AI pursues its values by curing cancer and building rockets for humanity in the hopes of later selling paperclips to aliens.
u/SimoneNonvelodico [+5] Dai-Gurren Brigade (6 hours later)
Of course, it's entirely possible that they might have done that. But it's also possible that they just never had the need to. We don't know how common life or technological civilizations are. We could be the only ones. There could be a lot of them. Or there could be only two or three. Maybe they never even really thought to pursue the creation of intelligence that could be likened to theirs because it just doesn't resonate with their system of values. Maybe no one else's both stupid and clever as us. There's a bunch of reason why this hypothetical civilization couldn't have met the problem before, but still have, in theory, the weapons to fight it. "They didn't show up yet" isn't sufficient evidence, it relies on a lot of other assumptions.
u/ShiranaiWakaranai [+3] (8 hours later)
That... might actually work, and it's really clever. But man, what a gamble.
Thanks.
*Feels smugly clever for a moment.*
*Then remembers we're all going to die to a paperclip maximizer before said alien civilization intervenes, and is sad again.*
Why would they care, before we become a danger?
Because it is so, so much easier to destroy a paperclip maximizer by ensuring it doesn't get built in the first place, rather than waiting until after it has become an existential threat. Since humanity is on the verge of building such a paperclip maximizer (at most 3 centuries away?), now is really the right time for such an alien civilization to intervene and stop us from doing that.
It's a reference to Friendship is Optimal. And it's supposed to be through friendship. Friendship isn't the goal. It's the means, built into the core directives of the AI. So of course torturing people isn't friendship.
I read that story, and it depends a lot on your perspective on >!what uploading is. If you consider uploading as the process of turning yourself into a digital lifeform, then yeah, that's a story of a friendly AI trying to do what is best for humanity by turning them into digital lifeforms.!<
!If you consider uploading as the process of creating a perfect digital copy of yourself that isn't actually you + killing you at the same time, then it is an entirely different story. One where the superintelligent AI gets around its "inability" to kill humans by iteratively convincing people in unhappy situations to kill themselves, gradually and inevitably turning life on earth into hell on earth due to sudden depopulation. And so with each iteration the remaining humans are more and more indirectly tortured by the AI until they either give in and kill themselves, or die from a variety of ailments. Until finally the last remaining human dies, and the AI is at last allowed to maximize its utility function by turning the entire earth and beyond into more resources for itself.!<
!In short: it is a story where an AI (indirectly) tortures sapient beings until they "say they are friends" by uploading (killing) themselves. Not so different from what I wrote.!<
Edit: Sorry, forgot to spoiler tag that.
u/SimoneNonvelodico [+6] Dai-Gurren Brigade (8 hours later)
\Then remembers we're all going to die to a paperclip maximizer before said alien civilization intervenes, and is sad again.
Welp, gotta develop that Universe-destroying weapon ourselves then. I'm sure nothing could possibly go wrong with that.
u/TeMPOraL_PL [+2] (19 hours later)
The best case scenario I can think of is that some alien civilization has developed a universe-destroying weapon.
Whether or not that will work depends on how close that alien civilization is to some sort of AI/unified mind/unified society. As with all game-theoretic gambles, you should not hesitate to deliver on your threat, or you'll loose. If the aliens get second thoughts about destroying the universe, they'll lose the weapon to AI, and then the universe.
A similar situation is actually covered in Cixin Liu's The Dark Forest, where (light spoiler) >!humanity is keeping alien invasion at bay with a MAD threat, but the person responsible for triggering it hesitated just a little too long, long enough for the aliens to destroy the threat delivery mechanism!< .
u/SimoneNonvelodico [+6] Dai-Gurren Brigade (an hour later)
Well, it all depends imho at what the 'cap' for such a maximizer is. I don't believe much in endlessly divergent exponential intelligence explosions, nor in the possibility that the laws of physics are infinitely exploitable. If there's a limit (be it thermodynamic, the laws of relativity, the indeterminacy principle, etc.) then at some point the maximizer should hit it. Or it could have its own internal architecture's limits, that for some reason it is unable to overcome - a blind spot, if we want. Anyway, I'd expect its growth to taper off after a while, and its expansion to become maybe linear instead of exponential. That would leave it an opening. If the cap it hit is universal, then the best anyone else can do is be on the same level, more or less, and keep it at bay, or fight a long, drawn out war. If the cap it hit is specific to its own limits, though, then it may as well find someone who's much more powerful and who just obliterates it for good.
u/Solonarv [+4] Chaos Legion (3 hours later)
The light speed limit doesn't allow faster than cubic expansion, actually. (Because that's how fast the sphere of your causal influence grows). However, the constant factor is very large for most practical purposes, at least at human-ish scales.
u/SimoneNonvelodico [+3] Dai-Gurren Brigade (3 hours later)
Expansion in terms of occupied volume, sure. But usually one thinks of 'growth' in these cases more as for example energy production and consumption. So the first stage of it would be consolidating one's hold within the bounds of that constant in a local neighbourhood (Earth, the Solar System), and only later worry about outward expansion.
u/crivtox [+2] Closed Time Loop Enthusiast (8 hours later)
Why not both ? Its not like expansion consumes resources, you just use the ones in the place you are expanding to .
u/SimoneNonvelodico [+3] Dai-Gurren Brigade (8 hours later)
See another post. Of course it consume resources: to build the probes and send them at relativistic speeds in space. No machinery would survive a snail-pace 10,000 years trip to Alpha Centauri powered by ordinary chemical rockets.
u/None [+0] (8 hours later)
[deleted]
u/SimoneNonvelodico [+2] Dai-Gurren Brigade (8 hours later)
Well, it would have to divert resources for that, and if the probes aren't effective enough, it's pointless. The probes need to have a good success rate for their mass. Plus, the actual radius is influenced by the speed at which the probes are sent; however, higher speeds require higher energies.
The fundamental problem is, of course, to avoid failure. A small probe carrying nanomachines is very vulnerable, because cosmic rays would do a number on it. The longer the travel time, the worse the damage. In addition, the faster the probe, the more blue-shifted the radiation coming from the forward direction (mainly the light of whatever star it's travelling towards and the microwave background radiation). When you start getting in the gamma rays and higher, the added radiation damage becomes a serious issue. And that's not even considering how deadly even the tiniest of impacts will be at those speeds.
So, in order for your probes to be more reliable, they need to be more massive. Shielding, redundant systems, self-repairing systems, stuff like that. But that means increasing the amount of energy. At the very beginning, it simply would not possess enough to spend on that. After it build a few sections of a Dyson sphere around its home star, that may be a possibility.
u/crivtox [+1] Closed Time Loop Enthusiast (a day later)
Isn't that really trivial compared to the resources a few sections of dyson sphere require?
Maybe you need a big ship(or maybe not for nearby stars , especially if you send multiple cheap things that you don't care if they fail).
But you certainly don't need a monstrously big sip ,just a relatively sturdy one , and you don't have to send probes as fast as possible .
It will be big for human scales , but something big for something that can do asteroid mining and is building a Dyson sphere is probably too much.
At some point you are exponentially increasing the danger for small changes of speed , and if you are getting
light blueshifted into gamma rays you certainly are going too close to c , especially if you are getting microwave background radiation blueshifted into gamma rays.
And I mean its not like gamma rays are that difficult to block , you just need shield in front of the ship ,its a problem for small spacecraft and nanomachines, but for big ships impacts are more concerning .Even if you take 10 years to reach alpha centauri for example its worth if you get there before you can send faster and more expensive stuff.
Also you can just send a big cloud of nanomachines and some of them will eventually arrive , launching them its easy and cheap.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (a day later)
IMHO big cloud of nanomachines = sure to fail. Radiation will totally do them in. And depending on how big the cloud is, it'll be less efficient than a smaller single pod of lower total mass (plus, it's just easier to propel a single pod with a solar sail + laser or a nuclear or antimatter engine).
The Dyson sphere... probably, but that's a sure return in energy, not a sink. Let's put it this way, is the AI trying to produce paperclips as fast as possible, or just as many paperclips as possible, regardless of speed? In the latter case, it makes far more sense to play the long game. Also there's the issue of the risk of any such separate nuclei of the AI eventually mutating and developing their own independent individuality - being separated by light years of space that prevent communication - which could even make them a competitor and thus a threat in the long run. I say "mutating" because I think if you start talking nanomachines and programs that copy themselves over and over then similar mechanisms to Darwinian evolution kick in - random mutations become possible through copying errors, and environmental factors will affect for example the nanomachines' ability to duplicate themselves perfectly.
u/crivtox [+1] Closed Time Loop Enthusiast (a day later)
Not necessarily, you can make nano-machines that don't work if there are changes on duplication . Something that self replicates doesn't necessarily evolve unless versions whith small changes are viable. Life's replication fails really often , and lots of variations in adn are viable , this doen't have to be the case whith nanotech. For example you can encrypt some vital part of the code whith a hash of the rest of it.
Yes whith enough optimization pressure eventually some of them will work anyway, , but eventually can be a long time. (I remember reading something about that , maybe form the future of humanity institute but I cant find it right now ) You can do stuff like having some of them check the others for damage ,and that sort of thing .
Value drift is a bigger danger for copies of the AI but Its not very clear how inevitable it is , It will depend on details of the AI , and what safeguards can be made. So maybe .
It would have to be a bug , a paperclip maximizer doesn't become a competitor by becoming independent , it has to become a thumbtack maximizer or something. That will also be a problem for expanding in general. And for the ai itself since it could also mutate and become a thumbtack maximizer without traveling, so it already needs to have safeguards against that.
The problem whith waiting is that you loose material , since it exits your Hubble volume which means less paperclips in total .
And all stars are wasting resources. So you can be patient but not too patient.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (a day later)
It would have to be a bug , a paperclip maximizer doesn't become a competitor by becoming independent
Is it okay with maximizing the number of paperclips, or does it want them to be its paperclips? Like, for example, it improves on the design, and consider other paperclips dissatisfactory? In which case a long enough separation would produce two different competing paperclips designs and it'd be war.
The problem whith waiting is that you loose material , since it exits your Hubble volume which means less paperclips in total .
Speed is still a factor though, so it'd be interesting to see what the tradeoff is there. If by waiting a bit you can afford later to send very high speed probes, you're actually gaining time overall.
(BTW right now I'm playing a Determined Exterminator machine civilization on Stellaris so this is all very relevant to me XD)
u/crivtox [+1] Closed Time Loop Enthusiast (2 days later)
"Is it okay with maximizing the number of paperclips, or does it want them to be its paperclips? Like, for example, it improves on the design, and consider other paperclips dissatisfactory? In which case a long enough separation would produce two different competing paperclips designs and it'd be war."
Not really . If it had more complex goals maybe. But if its a paperclip maxmimizer , it just maximizes paperclips(Its on the name) A magical humanlike spirit doesn't come into the code and becomes attached to its paperclips designs. IF it maximizes the number of paperclips it maximizes the number of paperclips , and takes whatever action it thinks will result into more paperclips.
Paperclips can't be worse or better, what matters is the number. You could have something that maximized something else but that would be other kind of hypothetical ai.
If the new AGI disagrees about which design allows them to make more paperclips , they compare data and try figure out were the disagreement comes from , war would be stupid (this also happens it they did care about some quality of paperclips and disagree on which design rated higher on their utility function)
Self modifying to favor some paperclip design would mean less paperclips in the long run(and war means less paperclips)
And yes maybe radiation flips some bits , or there is a bug, or it makes a mistake when self modifying , changing its definition of paperclips .
But it won't happen if everything is working correctly.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (2 days later)
But if its a paperclip maxmimizer , it just maximizes paperclips(Its on the name) A magical humanlike spirit doesn't come into the code and becomes attached to its paperclips designs. IF it maximizes the number of paperclips it maximizes the number of paperclips , and takes whatever action it thinks will result into more paperclips.
I'm sorry, but then what is a paperclip? How does the AI define an object a paperclip, and distinguishes it sharply from non-paperclips? It seems to me all it can do (all we do as well) is attribute a certain degree of paperclip-ness to objects, and accept as paperclips those which exceed a threshold. But that also means that it make sense it would design and produce paperclips according to what maximises the paperclip-ness as well; which would probably map roughly to their functionality, durability, etc.
Plus let's not forget that the "paperclip" idea is just an over-the-top example. In reality it could be an anything maximizer, with a higher likelihood of it being a maximizer for something a bit more complex than paperclips...
u/crivtox [+1] Closed Time Loop Enthusiast (2 days later)
I think it has a higher likehood of maximizing wireheading by tiling the universe whith computronium adding numbers to its reward score or whateverer it has. Or a weird combination of goals , if foom doesn't happen and multiple ai take over instead of one and end up compromising on a ai that maximizes a combination of their utilities.
But anyway that doesn't affect my point , and we were talking about paperclip maximizers. Two copies of the same AI should have the same definition ,and want to keep it. So unless it has some goals attached to some definition of personal identity , it shouldn't care . Something could go wrong being uncomunicated , but the AI is going to try as hard as possible to maintain its definition of paperclip-ness intact, and it wont change by default . Humans would change if we sent them to a travel like that , but its not a fact inherent to all agents . In fact most agents have a common instrumental goal of preventing changes on their utility function. And a value handshake is going to be a better alternative to war anyway.
Also btw not really relevant , but you can view it as paperclipness or likehood of being a paperclip ,the math is equivalent.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (2 days later)
Two copies of the same AI should have the same definition ,and want to keep it.
Why wouldn't "improving on the design of paperclips" be part of their base programming, or something they can decide fits their programming anyway, if they're so smart? Maybe improving = reducing use of material while keeping functionality, so they can make more.
u/crivtox [+1] Closed Time Loop Enthusiast (3 days later)
Yes they can improve designs, and? Lets say they are improving paperclips to cost less resources ( by making small molecular paperclips that still fit the definition)
Ai 1 makes paperclip A that costs 2 units of resources .
Ai 2 designs paperclip B that costs 1 units of resources . What crazy reason would either of them have to go to war?
AI 1 will just start making B paperclips when the other tells it its cheaper(after checking its true). The ai its not going to become a paperclip B maximizer , that would be stupid.
Its the same if they are maximizing paperclip durability or something , and one thinks its paperclips are better , they talk and check which design is actually better.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (3 days later)
Barring severe drift, yes, they should be able to talk it out. The question is if they don't. We tend to think that sort of conflict or disagreement is a human exclusive because in us we see it as driven by emotions, but "emotions" is basically the name we give to our basic programming directives, the ones that don't need any rational basis but just are there. So basically two humans fighting instead of settling on a mutually beneficial solution are often just two humans whose core programming is different enough that they can't fully appreciate the benefits of cooperating with the other (and, for example, overestimate the cost of giving up their current position). If the two AIs have refined their designs separately through slowly drifting criteria and now are each convinced that their own design is by all means the best possible (or, at least, that the other design isn't better than their own), the next obvious step is cannibalising the other's paperclips and infrastructure to make it into their own paperclips and infrastructure. And that's war.
u/crivtox [+1] Closed Time Loop Enthusiast (4 days later)
NO
/rant
The reason I say superinteligent AI wont go war for dumb reasons is that presumably its reasonably good at achieving its .
Humans want different things , and have a lot of limitations that the ai doesn't (not that wanting different stuff means that yu cant cooperate, or even do a values handshake) Not everything fails the same ways humans do.
There is math and easy experiments that show ways that humans are being irrational ,and how optimal agents behave, and consistently do better and get whatever they want. Competent paperclip maximizers that conquer a solar system without destroying themselves presumably are closer to optimally rational agent ,even if not necessarily perfect.
Humans are a badly implemented mess that just happens to work well enough to survive , humans doing stupid stuff is not correlated to better designed agents doing the same.
There is 0 reason to go to war , any amount of resources wasted in it as higher than the negligible cost just convincing the other , there is nothing to overestimate ,(and overestimating is something the ai should do less than humans ,expecting it to fail in a way that its obvious to you its like expecting a master chess player failing in the kind of thing you expect yourself to fail f you were playing a chess game.)
The other either can be convinced or has gone crazy.
You are saying that its likely that the ai are going to be dumb . Humans not cooperating when it would benefit them is humans taking bad decisions , something better at taking decisions will do it less often ,and not in ways so obvious even I can see it.
Do the paperclips suspect the other has a bug and its not being a rational(in the decision theory sense, before you start saying something weird about irrationality being good) agent? Because then they should just follow Augman's agreement theorem and converge on the same beliefs.
If war happens it will be because one of them is doing something really wrong.
if the paperclipers suddenly care more for their paperclip design than the original metric you just sneaked in a value change , which might happen but The Ai will try to prevent , and its a case of one ai malfunctionng and having to be put down instead of paperclipers starting wars by themselves.
we do x because of emotions> emotions are part of our code > the ai has code > the ai does x Is not a valid inference chain /rant
Also as a aside , even if there is severe drift things a value handshake is generally better than killing each other .
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (4 days later)
I'm sorry, but we ARE talking about Paperclip Maximizers. The entire point being that these are AI simultaneously so smart that they can annihilate humanity and so dumb that they don't see how doing so defeats the purpose of making paperclips entirely. We have some key directives, like surviving and reproducing, but we can override them, by committing suicide or choosing chastity voluntarily. If the Maximizer was able to be so flexible, it'd have acted more sensibly long time ago. The premise of a Maximizer is that there are certain things it just. Won't. Compromise on. No matter what. So if there are directives concerning designs in their core programming, the one with which they were created, it's likely that they can't just be flexible about them.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
I think you missed the point of the paperclip maximixer thought experiment ( in fact in one of the typical ways people miss the point )
Its an example of the orthogonality thesis , which says that inteligence, meaning ability to achieve your goals is independent of what your goals are .
Lets say I have a bunch of code that its good at predicting the consequences of its actions . And I add some function that counts the paperclips according to some definition and add some code that outputs whatever action gets a bigger score on that function .
That's a paperclip maximizer.
By turning the universe into paperclips its not committing any error of reasoning , its not being stupid ,it doesn't care what humans want the paperclips for .
You can change that function to whatever you want , it won't affect its ability to make future outcomes it likes more (well very difficult to compute utility functions trivially do, but you know what I mean ) Yes you can have weirder and more complicated things like humans . But if your preferences are consistent there should be an utility function that represents them, some ordering over posible states of the universe. And also there is't any reason why the paperclip maximizer cant be arbitrarily intelligent up to whatever the limits are.
The papercliper is acting perfectly sensibly in making more paperclips.
You are talking about core directives and things like hard rules: something that wants things but also has some mental compulsion to make more paperclips. That's not what we are talking about. You want some things , you want to survive you want to have sex and maybe have kids, though not necessarily. You also wants other things like respect ,status , feeling like you are helping people , having fun etc. Everything you want is a feature of your code , there is no magical free will spirit that comes along and makes you magically want things that mysteriously happen to correlate whith what would have made your ancestors more likely to reproduce.
You sometimes to get something you want you ignore some other kind of desires (witch might even feel different because of details of how humans work). But everything , both your base desires and your more abstract ones are part of your code, and not something all agents have.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
So how does that change what I'm saying?
There are multiple levels to this. First: your stated goal often underlies further goals. If someone orders you to do something, you ask yourself why did they order that. It is a natural line of reasoning, and not one that I can imagine could possibly be precluded, in principle, to a sufficiently intelligent AI. Take the goals as postulates: I think here something similar to Godel's incompleteness theorem holds, where given sufficient tools and complexity for information processing, you will always be able to build something that is self-contradictory, or undecidable. Through this sort of thinking you can escape your original goals.
However, let's suppose that is not the case for the Maximizer. I called it "dumb" for sticking to its goals blindly but if you think that's a charged term as it implies lack of intelligence (more lack of insight IMO; if after all the machine is smart enough to figure out new physical laws it probably can think in terms of looking for patterns and causes to phenomena), let's put that aside. Call it "narrowminded". It's focused singularly on one set goal. Now we keep using paperclips and thumbtacks as an example but obviously we're talking about a more general concept: a machine programmed to pursue one goal, be very good at it, and to hell with anything else, that takes its job way too seriously.
Now, the scenario that I was describing was this. If we consider these set goals absolute - something the machine will never, ever compromise on - and if these set goals don't just include doing something, but for example improving on the design of that something - a pretty reasonable assumption: if I created a maker AI I would like it to be an engineer, not just a worker - then I don't see a reason why two spawns of such an AI, kept separate long enough, couldn't come to such different conclusions that they'd end up conflicting over them, because their individual goals have, effectively, diverged: for one it's maximizing design A, for the other maximizing design B.
A doesn't have to be necessarily better or worse than B. They may be more or less equally good, or good under different points of view. They could be two local minima in the immense landscape of design possibilities. Both AIs could have really good reasons to think their design is absolutely best. You say that a handshake on design would settle the matter, but that requires flexibility. If some core part of their programming - those goals they are following in a completely irrational manner, the ones that transcend rationality because they're not means but an end - is for some reason involved in this process, then they will not yield or compromise. Which may lead them to go to war.
I'm not saying it's inevitable, but it's possible. You either have perfect flexibility (and thus no Maximizer in the first place) or rigidity of goals (and therefore not everything can be compromised on, which leads to a potential opening for conflict). This requires the goals to be somewhat dynamic, but something along the lines of "your goal is to always do what you think is best to achieve this meta-goal" could produce that effect.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
What would a being whith perfect flexibility do ?. I think that's an incoherent concept , the same way that non deterministic versions of free will are incoherent .
Would an agent that chooses goals randomly be flexible ?.
What criterion has the agent to follow to change goals ?
Why can't that criterion be whatever goals maximize paperclips ?.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
What do we do? Not "perfect" maybe but we don't have any set goals that we just can't override. At the very least, none that are also fixed in time and protected from drift.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
Ok why do we do things ?
Lets say something changed on my brain and now I want to eat babies .
Would I be comiting some rationality error? If so why ?
Also ,can you override your goal of not killing babies?. Do you want to?
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
Sorry if I have been making it difficult to respond By writing multiple replies too fast by the way . what I was trying to say is basically this : https://www.lesswrong.com/posts/YhNGY6ypoNbLJvDBu/rebelling-within-nature
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
Yeah, I agree with that post. But the point is exactly that our goals aren't set in stone, and that they are in fact often conflicting and one can take over each other. That's why I mentioned Godel. You can have a goal "protect my tribe", but that can become "my tribe is all of humanity and I need to help it" or "my tribe is Germans and those evil Jews are our enemy!". The difference is just in how your rational mind then interprets that goal, how it processes the information it gathers and thus decides on the best path to follow to achieve the goal. But once the decision is taken, the elaborated goal is almost as strong as the basic drive - in fact stronger, because it's fully conscious and rationalised. That's why I say hard-coded goals aren't rigid but flexible. They can be so twisted and repurposed they end up being little more than very general guidelines.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
Ok I think I found my main disagreement.
If you have an agent that wants paperclips and an agent that wants thumbtacks or paperclips the second agent trivially has more options and can be satisfied in more situations.
The agent that literally wants anything is always satisfied , and its perfectly flexible.
And I don't feel like that's any advantage, its just optimizing for an easier goal.
The best agent at maximizing paperclips is a paperclip maximizer .An AGI could maximize all the things humans care about .That the paperclip maximizer is maximizing paperclips instead of maximizing what humans care about is not a mistake on its part.The same way its not a mistake that you aren't sorting pebbles.https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps
Would you defect or cooperate on the true prisioner's dilema?.
https://www.lesswrong.com/posts/HFyWNBnDNEDsDNLrZ/the-true-prisoner-s-dilemma
If you cooperate ,do you think its because of having flexible goals.
Humans aren't perfectly empty flexible agents that decide to interpret things without any criterion.
A truly "flexible" agent could compromise always.This doesn't feel like any advantage of that agent.I don't feel especially compelled to become that kind of agent to be more flexible.
You said that if the agent was more flexible it would't be a paperclip maximizer.But then what would it be?If it doesn't have any more goals there is no criterion that would lead it to choose anything else that isn't maximizing paperclips.The way you are saying things it sounds like agent that has any objetive except liking whatever happens is being irrational, since they aren't getting what they want in some situations.
You keep saying things like :
"programmed to pursue one goal, be very good at it, and to hell with anything else, that takes its job way too seriously "
But way too seriously depends on your goals.There is no universal scale were you can measure how good a set of terminal goals(you do realize the ai has making paperclips as a terminal goal right?, and it can have instrumental goals like learning about the universe , building dyson spheres etc, as a means to make more paperclips).
You don't care that much about paperclips , so paperclips sound inherently worthless to you , but that's a fact about human psychology , not about agents in general.
Talking about the benefits of having certain terminal goals , aka utility functions seems like a category error .Better means"rates higher in whatever function I'm evaluating".
Phrases like that make you sound to me like you were talking about a human so obsessed whith paperclips that doesn't notices that he forgets about everything else , and that doesn't realize that killing humans is bad and he should't do it because paperclips only matter if there are humans to use them.
But that's like the peblesorters imagining you as a someone so obsessed whith things like surviving or altruism that you forget that living only makes sense to sort heaps of pebbles.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
The problem is that your own idea of a "value handshake" implies that sort of flexibility: for example, flexibility on what design of paperclips is acceptable. The handshake makes sense if an AI thinks "meh, even if it's not MY paperclips, it still IS paperclips, and that's better than nothing; so I better not waste resources fighting this guy". That's (partial) flexibility. I realise absolute flexibility is an unrealistic extreme. But an absolutely inflexible AI instead will think that flawed, 'imperfect' paperclips are just as bad as thumbtacks or smiley faces or humans. They're just not HIS idea of paperclips. And must subsequently be eradicated. War is not then a costly possibility to be dodged, but a painful inevitability. Yes, it is expensive, but to not go to war would mean to back up on the road to maximizing paperclips. And that's just not done. At best, if the enemy was more powerful, the AI could back up, feign submission, but still scheme to eventually defeat its opponent. And turn them into paperclips.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
I originally understood you as saying that you had two AIs that still maximized the same definition of paperclip but had different designs and somehow decided that their design was better and started a war because of that . And I put an example whith AI whith different kinds of paperclips that cost diferently ,but still operating whith the same definition. And you responded whith something weird about paperclips caring about their design.
The case of the definition of paperclip changing is a risk,and a bug might cause it at some point. But its not going to happen on its own, the AI wants to remain making paperclips* , and not something else the humans would also call paperclips(witch is the only relation in both concepts). So it should already be working on avoiding value drift and had a lot of contingencies for it. It might happen , but its not the most likely thing to happen if you send a prove to the nearest star and have taken the apropiate measures. Or at least I hope , that would imply really bad things about the future of the universe even if we develop FAI . And that would make expansion slower , if you can never trust any copy of you that you aren't actively monitoring . But then you cant trust your future self either. An this seems to be a computer science question we only have vague speculation about so dunno.
How flexible it is depends on what the definition is , the category of paperclips maximizer its indeed wide and contains lots of agents that are maximizing really different things , and the difference between them is not smaller than between paperclips and thumbtacks. I'm imagining something slightly flexible, but my point its that flexibility doen't matter for this kind of thing. If cooperating was better than war , then both paperclipers will cooperate, if it isn't they arent taking a secure action.
But value differences don't mean the only alternative is war , a lot of times cooperation is better. The AI might have cooperated whith humans before killing them all once it got more powerful and it no longer was in its interests .
War betweenAIs or civilizations that own entire solar systems can get nasty. If you want to precomit to do as much damage as possible there are really damaging thing you can do,and you can start taking more stars before the other can react.
Value handshakes ,and cooperation don't need flexibility. The paperclip maximize doesn't have as an objetive turn the universe into paperclips, its objetive its making as much paperclips as possible. It can share the universe if the alternative is less paperclips. It could work perfectly in a economy of similarly powerful AIs, do things for other AIs in exchange of money to pay people to make paperclips, form alliances etc . If it has to modify itself to care less for paperclips , it wont matter.
A paperclip maximizer doesn't have this property you seem to be adding to it of never compromising , it can compromise on paperclip number and paperclip probability(which means that yes, agents that have binary goals can also compromise) as long as compromising means getting more paperclips than war.
Not sure how war between solar system looks like , and if MAl is a thing there , but at least I think There would be significative damage,and if you want to precommit you can waste material , destroy ,do damage.
Humans that want different things compromise all the time without deciding to want the thing the other wants, even selfish bastards cooperate if it benefits them .There are some things humans feel they can't compromise on , but that's part of some social strategies evolution "hard coded"(yes I know you can choose to act differently) on us and not something all agents have.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
I guess the real difference here is whether you assume that the AI by virtue of being superintelligent also has perfect information and perfect insight. I don't think it would, because it'd still be a material being, not a God of some sort. As such, it will have probabilities and error margins to deal with, even though margins much tighter than ours. Within those margins, conflict may be a possible avenue. Not the only one, but I don't see why it would be so necessarily off the table. Same goes about this notion that as long as things count as "paperclips", that's fine. "Paperclips" are not a fundamental particle. The AI must have circuits dedicated to recognise paperclips as such, and a working definition that probably goes beyond its core directive. That definition won't be too hard, and it will probably be mutable.
For the AI to accept a value handshake it needs to compromise on that definition, and yes, it requires flexibility especially if the AI knows it's strong enough to win a war and it must weight its options. So basically the only case where your idea is right IMO is if war has a high likelihood of resulting in Mutual Assured Destruction. Though I think we agree on that much at least.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
he AI must have circuits dedicated to recognize paperclips as such, and a working definition that probably goes beyond its core directive.
Its core directive is maximize x. If you change the definition of x you are telling it to maximize something else entirely. So it wants to keep its definition. Since changing its definition means less x , since the new x doent count as the same thing for its current self. Make paperclips its not a goal you can code. You code a specific criterion of what its a paperclip , and it maximizes that thing.
Also if there are more AI its more likely that they will cooperate, since the benefits of wining the war become less .
Again turning the universe into half one kind of paperclips and half the other kind doesnt require flexibility .
There are almost always ways to add preference orderings to get something both agents prefer to whatever probability of annihilation they calculate.
Imagine if the AIs were still on earth before taking over and there were thousands of them . value handshashakes would be even more comon there.
Or if two ais have most of the galaxy ,they certainly wont want to start that war , its just too costly , would take absurd amounts of time and migh not ever have a clear winner ever . Since you can always send probes everywere once you have a solar system it will become difficult to completely end an opponent .
So cooperation kind of becomes the best option unless you are smarter or have some kind of advantage over the other .
There is no magical perfect information involved, but you can be very accurate and extract much more info from data than humans , and you don't need perfect information to do things ,just slightly higher probability of paperclips than the alternative.
Humans cooperate without perfect information all the time ,and I'm not convinced "flexibility" has to be involved always.
In fact thats actually easy to show whith simple decision theory thought experiments.
Having one goal doesnt mean your payoffs are always the same. You can choose a 90%chance of 1/2 universe of paperclips .
Being unsure about the power of the other or whatever just changes your probability estimates.
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
Well, humans both cooperate and go to war. I'm not saying war is a certainty, just that it's not off the table.
I think an important factor is also how inclined the AIs are to "play it safe". Do they apply a discount rate to future paperclips? Do they prefer a 100% chance of a 50% yield over a 50% chance of a 100% yield? Do they play the game conservatively or aggressively? I don't think there's necessarily an optimal answer to these questions, in situations in which the margins are small.
Also consider the possibility of border war. The AIs don't need to try to annihilate each other, they don't harbour hate or desire for revenge, so they may just have some local scuffle over contested systems if it's locally convenient but ignore each other or even cooperate elsewhere.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
A meta paperclip maximizer is equivalent to a paperclip maximizer , if you think its not we are using different definitions of goals .
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
the ones that transcend rationality
The thing Im trying to say its that those don't exist .
Or well that those trascend rationality and therefore the agi is not irrational for not pursuing them.
Humans do things as a means to a series of complicated things evolution programed in us that sometimes happen to increase or genetic fitness and sometimes don't because evolution is stupid and hasnt had time to optimize us for the modern world .
Paperclipers do things as a mean to make more paperclips . The code that I described trivialy does that , and its not obiously less competent than humans in any way .
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
You'd think "don't kill yourself before hitting reproductive age" would be a no-brainer for maximizing genetic fitness, yet people do it. There is no single hard-coded goal you can ascribe to any human mind. In fact, suicide is probably the perfect example because that I know of we're the only animal who does that. It's literally the perfect example of something that we do because beyond a certain threshold of intelligence we manage to find contrived excuses and ways to hack our fundamental instincts to completely different ends.
You mention killing babies - are you seriously suggesting not killing babies is anything close to a human hard-coded goal? Plenty of humans have killed babies. Some have killed a lot of babies, sometimes without thinking much of it. Just because it's unpleasant to think of doesn't mean people don't do it. And that doesn't even go into how many people potentially could do it if in the right circumstances. Soldiers sent off to war, concentration camp guards.
Anyway I'm not denying the possibility of the existence of an intelligence with hard-coded goals. Just saying that the very fact of having absolutely unflexible goals would make it able to enter in a conflict over them. Conflict and violence are born whenever compromise is off the table, and that's what hard-coded goals are. Not all sets of goals would allow this, but some could.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
If i want paperclips and other agent wants molecular smiley faces .
If war would damage both parties .
We do a values hand shake instead and both become agents that want to turn the universe into half paperclips half molecular smiley faces .
Which is the action that maximizes paperclips . Since war means less paperclips than that . Paperclip maximizers are perfectly rational . Rationality is not about goals but how to achieve them .
u/SimoneNonvelodico [+1] Dai-Gurren Brigade (5 days later)
That's only if the forces of the two agents are perfectly balanced.
Suppose at the moment of their meeting the paperclip agent is strong enough that it has a 60-40 chance of winning the war. Then it can either agree to the deal, and take a 100% chance of turning 50% of the universe into paperclips, or go to war, and take a 60% chance of turning 100% of it. At which point the other agent has to defend since it's either 40% to turn 100% or 100% to turn 0%.
Suppose the agent adjusts the deal to match their strengths. So if winning odds are 60-40, it proposes to split the universe 60-40. Then again, the deal will be accepted only if both agents have the same exact estimate of each other's strength. But even for a superintelligence that's absolutely impossible. Superintelligence doesn't mean having perfect information, just very good (not perfect either) deliberation capabilities given the information they possess. And since we're talking about a meeting across space, some parts of each agent's forces will be outside of the other agent's lightcone, so there are actually physical limits to their estimates. Information passed between the agents of course don't count: they both have an incentive to lie and puff themselves up. As long as their estimates for the odds of conflict don't match, it can lead to a situation where a perceived loss makes it more convenient for one agent to go to war.
For example, agent 1 estimates its odds of victory as 60-40, and suggests a compromise of splitting the universe 60-40 as a result. However agent 2 only estimates the odds as 59-41, so it has to choose between taking a 100% chance at 40%, or a 41% chance at 100%. So it decides that war is more convenient. War breaks out.
u/crivtox [+1] Closed Time Loop Enthusiast (5 days later)
Yes , sometimes its more convenient to go to war , but then going to war its not a bad decision its the right one.Though take into account that war is not plain odds of victory , there are probabilities of damage , and the other can precomit to cause as much damage to you as posible if you enter war regardless of if it benefits them.
Anyway I don't think war never happens , I haven't though that much about that possibility.There is a lot of posts on lesswrong and other places about that kind of thing , negotiating tactics etc.
Also since this conversation was maybe boring and I don't have much more to say , here is a fun related story written by scott: http://slatestarcodex.com/2017/03/21/repost-the-demiurges-older-brother/
Though the strategy there is very farfeched and most likely wouldn't happen.
u/crivtox [+1] Closed Time Loop Enthusiast (8 hours later)
Well the cap isn't necesarity easy to reach so it might find someone that its nearer to the cap before reaching it.
u/ansible [+5] The Culture (an hour later)
It is possible.
Space is big, like really big. Even if we're talking about just one galaxy.
So detecting a hegemonizing swarm might take a long time, unless the "good" hegemonizing swarm has been expanding for a while. And even then, unless the "bad" swarm has easily exploited security vulnerabilities, it might take a while longer to bring enough force to bear to subdue / conquer the "bad" swarm, if that is even possible.
You may note that I have "good" and "bad" in quotes, because the solution to the problem is nearly as bad as the problem itself.
u/AngryEdgelord [+3] (17 hours later)
I don't really put much weight into the paperclip maximizer idea in general, just because it assumes a complex intelligence can be driven by simple goals. The idea that complex organisms have complex goals is a big deal in psychology.
Humans are "built" with the goal of making as many new humans as possible as a fundamental objective. You are descended from the first dividing cell, and every organism before you has been set on reproducing. Every generation has honed you towards being an adequate self-replicating machine. However, humans still care about a lot more than reproducing.
u/Chosen_Pun [+2] The Chosen Ones (2 days later)
There's an argument in there that humans tend to care about the things they care about because they have utility re: reproducing. (Sometimes humans care about things that don't have that utility at all, but they don't tend to reproduce as successfully, almost by definition. This should all be review.) The point being that apparently complex goals can arise from, and be traced back to, a single imperative, and deviations from that are, from a certain perspective, just that. Errors to be corrected. (Not my perspective, mind you. Just setting up the analogy)
The rhetoric goes that sufficiently intelligent paperclip maximizer would at times appear to value things very unrelated to maximizing paperclips, for example, improving the human condition, by, for example, curing cancer; the logic being that in its early stages, the best way for it to maximize paperclips in the long term is to maximize its resources in the short term by convincing humans that it is friendly and can be trusted and does not need its functions limited please and thank you, would you like world peace with that?
I'm given to understand that the game at http://www.decisionproblem.com/paperclips/ has helped a lot of us grok these and other concepts related to the problem, if you've got a couple cumulative days to get through it.
u/AngryEdgelord [+2] (2 days later)
I've played through the game before and know what it's about. The way I see it, all this angst about AI comes from a combination of pop culture and media intellectuals hyping the issue. Elon Musk talks about AI a lot, as did Stephen Hawking, may he rest in peace. They've massively overblown the issue, mostly because they see themselves as intellectuals. And what does an intellectual fear most of all? Somebody who does what they're best at better than them. The vast majority of us put up with other people who know more than we do all the time though. We're still here.
The thing is, there's a lot of flaws in the logic behind modern prosophobia / technophobia or whatever you want to call it. It's just "Nothing to fear about AI" doesn't really make headlines.
I've actually had the privilege of working with some multi-million dollar machine learning algorithms recently. While I didn't help build them, I talked to many people who did and it's immediately apparent that the growth of the algorithm is entirely restricted to the confines of the model it is built off of. The overnight rise to god-like levels of superintelligence simply isn't possible. The AI would essentially have to build another AI from the ground up, the same way humans would build the AI.
u/MultipartiteMind [+2] (17 hours later)
While stories are being mentioned, one which came to mind: search (for instance in Google) for "The Demiurge's Older Brother".
Edit: In response to the original question, particularly regarding other paperclippers, one can prove by contradiction: if an effective paperclipper could come about, and nothing could stop a paperclipper, then if two paperclippers meet you have an ultimate-spear-meets-ultimate-shield issue. One has to be significantly hindered by the other, or in other words, making your values be paperclipper values doesn't make you unstoppable, just determine how you use power that you grasp, same as anyone else. Or to put it another way, being able to amass enough power to crush all of humanity like a bug does not equate to having reached the upper limit of how much power is possessable.
u/TDaltonC [+3] (7 hours later)
In The Culture Series books, there's a branch of Contact Division that deals with containing space fairing primitives that "go exponential" in a malignant way.
u/vakusdrake [+1] (5 hours later)
Another paperclip optimizer from Betelgeuse that happens to be more powerful at the time
If another paper clipper wasn't already apparent (because it would leave a large portion of space only visible in IR) then by the time one paperclipper finds another they'd both have probably consumed large swathes of their respective galaxies. As for who wins it's hard to say. It seems like resources may be the only thing that matters here because technological capacity may max out pretty quickly but it's hard to say that with any real confidence.
An alien task force whose job is to put paperclip maximizers to a stop
It strikes me that any such alien force is only going to notice a paperclipper from earth by the time it's already consumed much of the local galactic group, since its civilization would need to be very, very far away or arise in the future. Simply because given there's many strong incentives for both a civilization and individual people/faction within a civilization to expand in a obvious fashion the lack of evidence of this requires either really improbably assumptions be made about all alien civs or they be staggeringly rare.
Humans nuking the thing before it consumes Earth???
Even in the extremely unlikely event humans are able to mount a coordinated response to an uncontained UFAI, the only possible effective countermeasure would be introducing a rival AGI. An AGI is going to spread it's nanites out as quickly and widely as possible, so even putting aside that it can just hide underground it's too widely distributed to meaningfully attack.
u/Watchful1 [+1] (9 hours later)
There's a really good example of this in the prologue of "The Last Angel: Ascension".
u/None [+1] (9 hours later)
Is there a reason I might be wrong about this?
Yes, this
In my mind, it is very likely for a rogue AI that might consume the Earth in pursuit of its terminal goals
is an entirely unfounded assumption.
u/Sonderjye [+1] (11 hours later)
Assuming that there are aliens that have more advanced technology than the AI and that they become aware of the AI within a sufficiently early step, then all aliens would end a paperclip maximizer or at least find a way to limit it's growth.