scrape_r_rational/data/cache/5pmteq.md

[D] Von Neumann probes seem like a very dangerous idea

Post:

For those unfamiliar with Von Neumann probes: https://en.wikipedia.org/wiki/Self-replicating_spacecraft#Von_Neumann_probes "A von Neumann probe is a spacecraft capable of replicating itself."

Regardless of why you would use a self replicating probe (exploration, seeding, even extermination of other life forms), if you lose control of their self replication abilities then you are going to fill the Galaxy with an endless tide of your Von Neumann probes. And eventually that swarm is going to grow and envelop everything, including the place where it was created.

Assuming you create a single probe for every star in the milky way galaxy, that is 1 trillion chances for something to go wrong with your probe and set off a chain reaction of out of control self-replicating probes. And it only takes one occurrence.

I think it would be incredibly irresponsible and dangerous for humans to ever create Von Neumann probes, especially if the only purpose of doing so is to explore the galaxy. That is a very minor payoff for a galaxy ruining level of risk.

When I see these probes in any works of fiction I can't help but think the civilization that creates them is either dumb or careless.

Comments:

• u/pixelz:

  On the contrary, we must launch such probes at the first opportunity to have any chance of defending against competitor swarms.
  

  • u/cjet79:

    That wouldn't be defense. It would be annihilating yourself first before anyone else can do it.
    

    • u/Law_Student:

      Just as easy as making a probe that uses whatever material to replicate would be making a probe that uses material selectively, you know. It would also be possible to make machines that preferentially replicate by disassembling out of control self replicators.
      
      There is also the opportunity for safety features like parent probes checking their children for errors and repairing or destroying them if necessary.
      

• u/xamueljones:

  You seem to have the idea that any error in the code whatsoever will eventually lead to variation in the probes. Then the variation will allow for evolutionary pressures to take effect, resulting in wildly divergent probes including ones akin to a "grey goo" scenario.
  
  The flaw I think you have in your reasoning is that not all replication permits evolutionary processes. There are multiple requirements:
  
  * Reproduction - The organism can create another alike itself. This is the whole premise of the Von Neumann. Check!
  
  * Heredity - The offspring resembles the parent. It is desired for the probe to be identical to the parent. Check!
  
  * Selection - The less fit offsprings will die out. Not Check! All probes will be making more of themselves, and there is very few factors to 'weed' out the unfit probes. Maybe reproduction speed will play a role, but by the time we have a theoretical probe which is better than the existing probe, we'll already have (at least) millions of the original design still spreading. While speed plays a role, there isn't anything to eliminate the 'inferior' versions if it's still capable of spreading.
  
  * Variable Fitness - Some offsprings are better at surviving/reproducing than others. This is the main flawed assumption. You are saying that if there are any errors in the reproduction, then evolution will result. But there can only be competing strains of probes if the mutation is somehow better at spreading through space, faster at reproducing, reproduces more, or is in someway *better* at something than the original type.
  
  Evolution is not a magical process that generates working lifeforms out of disorder in a finger snap. It is a very gradual process which develops new adaptations with very little change between the parent and offspring. Making a probe more efficient or faster is (most likely) a very hard problem the best human minds will be trying to solve. Complex adaptations which do something outrageous like taking advantage of unknown laws of physics or warp gravity to travel faster can't spontaneously appear out of nowhere since it will require multiple adaptations which in the process will likely be a detriment to reproduction.
  
  Sexual reproduction is an evolutionary strategy to *speed* up evolution and it still takes hundreds of generations for even the most minor adaptations to spread. The probes will be more similar to asexually reproducing species which are hundreds of times slower at evolving. Furthermore, the probes will take centuries to millennia per 'generation' traveling between stars.
  
  Any variability in the probes will overwhelmingly be more likely to be a detriment or be meaningless.
  
  Finally, this is something that can be tested beforehand, via computer simulations by deliberately introducing errors to the code and see what viable results can occur. Then we can introduce self-correcting redundancies to parts of the code most likely to go wrong as well as general error-checking mechanisms to prevent the errors in the first place.
  
  TL:DR - Evolution requires very minor errors which can lead to numerous viable mutations. Evolution is a process which in this case is already unlikely and we can prevent it to arbitrarily close to zero likelihoods.
  
  Sorry for having such a long post. I kept coming up with more to say!
  

  • u/cjet79:

    I can't see how there isn't a selection effect.
    
    You send out 10 Von Neumann probes. Half of them stay healthy and 'reproduce' passing on their healthy systems to their 'offspring'. The other half encounter errors. 4 of the 5 that encounter errors have an effect that makes them less fit or completely unable to reproduce. Those 4 lines disappear. The 5th one has a random error that causes its communication systems to fail, so it can now send probes to systems that have already been explored. This is a huge jump in fitness compared to the 5 healthy probes which have to send their probes to unexplored systems.
    
    There is a selection effect against probes that can self diagnose errors and shut themselves down, and a selection effect against probes that do not re-explore the same solar systems as other probes.
    
    The probe doesn't have to take advantage of some exotic laws of physics, it just has to have a bug or error in the part of the probe that tells it to stop reproducing when its job is done.
    
    Remember fitness is a function of how fit an organism is to reproduce. A sterile body builder is less fit in an evolutionary sense than a a vial of reproductive fluid.
    

    • u/xamueljones:

      Yes that's correct, but what I'm debating about is how easily such an error can occur. If we build in redundancies and self-correcting mechanisms, errors become very unlikely.
      
      However, let's say there are so many probes that even unlikely errors do appear. The code can be structured in such a way that there needs to be multiple steps to permit the creation of more probes. If the odds of an error for each step is 1%, then just by requiring 6 separate steps to allow for unlimited replication the odds drop below 1 trillion.
      
      Evolution doesn't rely only on errors/mutations. It needs *useful* mutations that can appear with only a small change.
      

      • u/cjet79:

        >but what I'm debating about is how easy such an error can occur.
        
        It sounded like you were debating whether there was a selection effect, specifically when you said this:
        
        >Selection - The less fit offsprings will die out. Not Check! All probes will be making more of themselves, and there is very few factors to 'weed' out the unfit probes. Maybe reproduction speed will play a role, but by the time we have a theoretical probe which is better than the existing probe, we'll already have (at least) millions of the original design still spreading. While speed plays a role, there isn't anything to eliminate the 'inferior' versions if it's still capable of spreading.
        
        I'm assuming since you didn't debate me on those points you accept them. 
        
        So that means evolution requires more than you list above. In addition to **Reproduction, Heredity, Selection, and Variable Fitness** you also say there is a need for mutation.
        
        And I should be clear on this point: evolution does not require that mutations be beneficial in general, it just requires that mutations happen. Biological evolution mostly has fitness reducing mutations. All that matters is that sometimes it has fitness enhancing mutations.
        
        All that you are really arguing is that the mutation rate will be low enough that we won't have to worry. And my response to that is to remind you that we are talking about 1 trillion stars in the milky way galaxy, and possibly hundreds of trillions of celestial bodies that a probe might be able to visit. Those numbers you give of 1% chance with 6 separate steps. Its actually exactly a 1 in a trillion chance of all those 1% things happening. I did the math elsewhere in this thread but with 1 trillion attempts at a 1 in a trillion chance there is a 64% chance that the thing will happen. And that is playing unfavorably with my numbers. Because one of the errors that might happen is that one of those 6 separate steps starts getting skipped. And when I refer back to the danger of mutation and selection earlier this is what I'm referencing. A probe that only has to pass 5 tests is more fit to reproduce then a probe that has to pass 6 tests.
        
        Let me ask you some separate but related questions:
        
        1. Do you think it is possibly to build an unhackable probe. Unhackable in the sense that an intelligent agent knowing everything about the probe could not alter it in any series of steps that would cause a grey goo scenario?
        2. Your banking software is launched into outer space. It must continously operate for 1000s of years in the harsh radiation environment of space, and then at the end of 1000 years it must copy itself into a new physical medium. A physical medium that your banking software created with the same code that needed to be running perfectly for 1000's of years. Do you still trust that banking software with your money? If yes, how many iterations of this process before you don't trust your banking software (100 iterations, a million iteration, a trillion iterations, a trillion trillion iterations)? 
        3. Do you worry about other existential risk problems like GAI?
        

        • u/xamueljones:

          Okay, I was a little tired when I made my earlier reply so I apologize if I sound like I was switching topics from whether or not there is evolution to talking about errors. Furthermore you seem to believe that I don't care or believe the Von Neumann probes are a serious threat. What I was trying to argue is a solution to the hypothetical problem, not that there isn't a problem.
          
          I agree it's a bad idea, but I believe it's one we can cautiously work with rather than an idea which is so dangerous that we shouldn't do it at all.
          
          Just to reiterate, the basis of my argument boils down to saying evolution requires beneficial mutation. I agree that evolution can theoretically act on the probes and cause selection pressures which result in a grey goo scenario. However, what I'm trying to say is that such an outcome can be planned for and prevented from occurring.
          
          The idea of multiple tests was a bad one, so let me try a different tack.
          
          I believe it's possible to design the probes to make evolution difficult to occur. I'm imagining that we can design the probes in such a way that any minor change results in failure of the probe rather than a modified version.
          
          The way DNA is designed, it often results in a viable life form because the 'space' of life is dense aka all life has very similar DNA.
          
          We can create the design/code of the probes that any minor change (or even major ones) results in absolute failure. With a 'fragile' design, we don't have to worry about competing strains.
          
          > Do you think it is possibly to build an unhackable probe. Unhackable in the sense that an intelligent agent knowing everything about the probe could not alter it in any series of steps that would cause a grey goo scenario?
          
          I'm not sure. This question is a little too vague for me. It depends on how the agent can interface with it as well as how smart it is. If it has very few options to deal with the probe, then the probe can be unhackable. But if the agent is extremely intelligent and it can put the probe on a workshop table and use any tools it desires for as long as it wishes, then I don't believe the probe is unhackable. However evolution is not intelligent, so I believe we can make the probe unhackable to evolution.
          
          > Your banking software is launched into outer space. It must continuously operate for 1000s of years in the harsh radiation environment of space, and then at the end of 1000 years it must copy itself into a new physical medium. A physical medium that your banking software created with the same code that needed to be running perfectly for 1000's of years. Do you still trust that banking software with your money? If yes, how many iterations of this process before you don't trust your banking software (100 iterations, a million iteration, a trillion iterations, a trillion trillion iterations)?
          
          First off, I definitely believe that such a banking system can theoretically exist. The question then boils down to how I can verify a given banking system to be as good as the theoretical version. I would want to see explanations for how it can survive the harsh radiation and what sort of precautions it would take against errors such as redundancies, prevention, and so on. It would take a lot of pre-testing and research before I would trust such a system. But if I need to store my money somewhere for a long time and I can't check on it in the meantime, then I would trust the banking system.
          
          > Do you worry about other existential risk problems like GAI?
          
          Yes.
          

          • u/None:

            Furthermore, any entity intelligent enough to hack a probe into a universal grey-goo machine... is probably smart enough to build a universal grey-goo machine, smart enough to know that they're hacking it into a grey-goo machine, and thus for whatever reason want to build a grey-goo machine. The probe is basically irrelevant at this point.
            

• u/hoja_nasredin:

  I see little problem with it. We the living beings are already self replicant robots.
  
  Nothing much worse can come out of making it on a galactic scale.
  

  • u/cjet79:

    If you have no preference for your own form of self replication, then sure there is no reason not to create another form of self replication that will replace you.
    
    Your indifference between grey goo vs human civilization is probably a very unique preference, and I don't think any space agency would put you anywhere near a Von Neumann probe project if they knew you had those preferences.
    

• u/callmebrotherg:

  worst case, you send in a pair of Quality Assurance bots with every probe; the QA are tasked with analyzing each bot (both new and old) and destroying anything that falls outside of acceptable parameters. It is possible that the probe could go haywire, and even possible that one of the QA bots could fall outside of acceptable parameters, but incredibly unlikely* that both QA bots would do so at the same time. 
  
  *I'm sure that somebody could do the math, if not now then once we actually have the programming written up. If the odds aren't low enough, then just add more QA bots.
  

  • u/cjet79:

    1. If you make the bots too cautious about reproducing they may not reproduce at all, or they might die out. 
    
    2. Part of the danger of Von Neumann probes is that they essentially have a genetic line of parentage. When they reach a new planet and replicate themselves their "child" may be more likely to carry over any mutations in the code or the construction. In this scenario you have to not just insure that a few random mutations will lead to a bad scenario, you have to insure that thousands of mutation steps won't lead to a bad scenario. The more cautious you are in eliminating certain lines of reproducing Von Neumann probes the more work you create for all of the remaining lines of Von Neumann probes. In the worst case scenario you have a single "genetic" line of Von Neumann probes that might have to stay safe through a trillion 'generations'.
    
    3. You are ultimately fighting against evolution and selection effects. Cautious probes will die out. Probes that first sustain damage to the QA part of their coding will be more likely to live on. Probes that have out of control self replication are far more likely to live on than their reproduction limited peers.
    

  • u/Flashbunny:

    1 trillion is a very big number - you'd need an awful lot of QA bots, given the increased propensity for data corruption in space.
    

    • u/Law_Student:

      Undetected data corruption causing insidious changes to programming isn't really an issue. Computer science has good solutions for detecting and either fixing corruption or destroying the corrupted data if it's so badly gone that error correcting codes can't save it. You can read about things like hamming codes and checksums and RAID arrays for some different approaches to the problem.
      

• u/Anakiri:

  ...Your Von Neumann machine became a threat to life? How? Why did you give it protection against re-entry, oxygen, liquid water, its own weight, and missiles? Why did you give it the ability to track and intercept spacecraft that can manuever more than ten million times as often as the rocks in freefall it should be targeting? Why did you leave any possible optimization for evolution to exploit by breaking your rules - that is, why isn't it already at a local optimum that is far away from the dangerous global optimum? The best possible robot arm doing everything by hand is not a grey goo doomsday scenario. Why did you make it reproduce using a mechanism that *even in principle* could ever be dangerously fast? It barely makes a difference whether a Von Neumann probe copies itself in a second or a century!
  
  Maybe don't do any of that. Make a slow, fragile, dumb thing that can't evolve into grey goo in ten thousand generations, let alone thirty. Then use error correction systems anyway - but don't rely on them to the exclusion of sane engineering. Your problem seems to be that you made a galaxy-killer with a limiter, instead of designing a probe for the mission. Don't massively overdesign stuff.
  

• u/TBestIG:

  Set it up so there are multiple backup systems protecting against mutation. Ideally you'd get to a point where it's almost impossible for something to go that wrong.
  

  • u/cjet79:

    Almost impossible is an unacceptably large margin of error when you only need 1 probe in a trillion for everything to blow up in your face. If you have a 1 in a trillion chance of something going wrong, and you have 1 trillion cases then you have a 64% chance of things going wrong. 
    
    To me "almost impossible" would mean something like 1 in a million chance. I'd just call 1 in a trillion chance of something happening 'impossible' with no qualifiers. You need to be able to say it would be impossible for the probes to go wrong just to even the odds.
    

    • u/Izeinwinter:

      Biological replication is not a good predictor for designed systems. DNA is an imperfect information transmitter because lines of decent where it was not became static and selected against. 
      Evolution acts via selection over variation - it is perfectly possible - and in fact sort of trivial for anyone that can build a self replicator at all to build them to just never vary at all. Mutations and variation will simply not happen to a line of decent that uses quadruple redundant information storage and defaults to sterility in the event of data corruption.
      

    • u/TBestIG:

      If almost impossible means one in a million and you have several billion of them, it's not 'almost impossible' anymore, it's very likely.
      

• u/LiteralHeadCannon:

  What if you deliberately "neuter" such probes, giving them the relevant skills to survive in uninhabited space but not the relevant skills to win a fight with your own military?  In order to wipe out your species, they would not merely need to become defective in the sense of "forgetting their loyalty to you", they'd need to become defective in the sense of "undergoing an intelligence explosion", which is obviously always a problem.
  

• u/luminarium:

  I wouldn't really worry about that. In the 100 years it takes the probe to get to Alpha Centauri, our technology will have advanced to be *far* better so that any probes we send out later will be much better at replicating and outcompeting, and we'll be spreading a lot more of them to other worlds, and we have the much larger industry, if it comes to war we'll win. And the thing about these probes is that any planet with a global net to catch arriving probes will have no problem getting rid of them, and we'll probably have that technology before the probes ever get to another planet, much less cover the planet and get back to us. 
  
  As an analogy, worrying about these probes is like worrying that compound growth in the stock market is going to make a regular person richer than the rest of the world combined. No, the rest of the world's wealth is growing at the same pace (and often at a faster pace).
  

• u/None:

  Load the probes with a governing AI that manages the replication. Or better yet, load the probes with uploaded minds *and* AI. Although at that point, the line between "probes" and "colonizing starships" starts to blur. 
  
  I'm pretty sure any self-respecting civilization would at least load a governing intelligence to prevent such a scenario as you described, and it wouldn't even be that hard once they have the tech to create such probes. 
  
  Plus, they may have other duties besides exploring. Seeding the galaxy with life, preparing planets for colonization, contacting and uplifting other species, scientific research, etc. Why send out multiple probes to do each thing when you can send out one that can do all? 
  
  A good way to reduce malevolent variation is to send the probes out in "swarms". Packs of a dozen or hundred each, that would check themselves and each other for discrepancies. The chance that a majority of them will experience an error that is both malevolent *and* prevents error-correction rapidly approaches zero the more probes there are in a pack.
  

• u/DCarrier:

  Encrypt the code. If it messes up a bit while copying, then when it tries to decrypt it it will get gibberish. And store the encrypted and decrypted code in very different ways, so there's no way it can accidentally copy the decrypted code.
  
  I think the bigger problem is something more intentional. Say you want a way to reprogram the robots after they're built. Then they can be reprogrammed by a hacker.
  

---