scrape_r_rational/data/cache/2oub3w.md

What, in your opinion, is the most threatening existential risk we currently face?

Post:

Exactly what it says on the tin. I'm interested in people's opinions here. I imagine that there will be a disproportionate amount of people citing AI, just because, well, this is /r/rational, but I'm fairly confident there will be a significant minority with something else in mind. (Frankly, if there isn't, I'd be worried about echo-chamber issues, even if AI actually is the most threatening thing out there, because really, the likelihood that practically everyone on a subreddit would come to the same conclusion independently is just too small.) I should point out that I myself am uncertain on this topic; AI certainly does seem threatening, but depending on your time estimate before it arrives, it's possible there might be more immediate problems we're facing. I'm just not sure. So... comments?

Comments:

• u/xamueljones:

  "What's the status of the project?"
  
  "Not very good. I don't understand why all of the civilizations keep self-destructing before they reach [Type I](http://en.wikipedia.org/wiki/Kardashev_scale)."
  
  "Are they self-destructing in the same way?"
  
  "That's the annoying part. I anticipated certain failure points such as the environmental, technological, and biological disasters, but they keep finding a different way to destroy themselves. The latest one failed due to poor social engineering. They ended up making everyone happy and in love with each other. But without any ambition, they stagnated and their birth rates fall so fast that they went extinct within a generation. It's enough to wear on my patience."
  
  "Maybe the problem isn't with your methods, but your efficiency."
  
  "Sir?"
  
  "Instead of trying to anticipate everything, let them learn how to anticipate the future and notice potential failures instead of having them 'naturally' evolve into an advanced society."
  
  "Hmm, that could work. If I let them do the work for me, they would be able to notice and prevent the problems occurring at the micro-scale, allowing me to focus on managing the macro-time scale. That will require teaching them rather advanced skills of thought at an unusually early stage. Maybe if I nudge one of their scientists into disguising the lessons as one of their popular works of literature....."
  

  • u/Kishoto:

    Just so we're clear, is this a reference to the Bible?
    

    • u/xamueljones:

      No. I just guessed that since so many people on this subreddit had read or heard of [Harry Potter and the Methods of Rationality](http://hpmor.com/), people would get the reference of learning about rationality skills through a rewrite of a popular book. But I probably made it too vague.
      
      The story is meant to say that we need to anticipate a *lot* of possible ways we can fail, to avoid extinction.
      

      • u/None:

        The reference was pretty obvious.
        

• u/alexanderwales:

  Rogue nanotechnology, super disease (engineered or evolved), and nuclear armageddon all get my vote over AI. The thing to worry about with AI isn't *just* that AI gets developed, it's that:
  
  1. AI gets developed
  2. It's able to recursively self-improve
  3. It goes rogue
  4. In a way that we can't stop
  
  I don't even know which of those points to be most skeptical about. When people talk about being afraid of artificial intelligence being developed, it's always the nightmare scenario of an artificial intelligence so advanced that it can talk people into anything or spin up new technologies on the fly. I'm not saying that I don't take it seriously, because I do. But in terms of the actual probabilities ...
  
  Well, look at the other scenarios. As an engineering challenge, self-replicating nanotechnology is *tough*, but I don't think it's as tough as hard AI. Same goes for engineering a bacteria or virus capable killing large swaths of humanity to the point where we can't bounce back. And nuclear armageddon only needs the wrong person (or people) to get into power.
  
  Maybe I'm just disillusioned by having read *The Singularity is Near* at an impressionable age, which had some dates that made the future seem a lot nearer than it really was, or maybe I've just worked on too many software projects to have much confidence in one so vastly exceeding expectations.
  

  • u/coriolinus:

    What, precisely, would you consider the difference between self-replicating nanotech and genetically engineered bacteria to be? 
    
    Grey goo is a classic SF fear, but it doesn't seem particularly plausible; in the end, these things need to power themselves somehow, which significantly reduces the stuff they can realistically eat / self-replicate from. There needs to be an energetic chemical pathway from raw materials to more of themselves, and bacteria have already taken up all the easy examples of those niches.
    

    • u/eaglejarl:

      Grey goo doesn't have to eat everything, though. It only needs to eat enough of the energy reserves that we can't sustain sufficient energy-harvesting to maintain civilization. Nanotechnology that ate fossil fuels would be an example.
      

    • u/alexanderwales:

      Basically what /u/eaglejarl said. I don't think it's out of the realm of possibility that a man-made nanobot (or engineered bacteria for that matter) would be more effective than something cobbled together by evolution. And they only really need to be good at deconstructing one specific thing - an engineered bacteria that was good at eating through trees would be nearly as deadly to humanity as one that could eat through anything, because it could cause a total ecosystem collapse.
      

• u/comport:

  Here are my current fears, in order of already happening to far fetched:
  
  **Economic Ecological Apocalypse**   
  Food instability driven by changing weather patterns, water shortages exacerbated by groundwater pollution and hydroelectric projects, the collapse of industries that rely on cheap consistently available oil, and climate-enabled health crises gradually worsen until all of the small factors combine and the global economy collapses in a flashpoint, marked by superpower proxy wars for control of [rivers, pipelines, shale gas] and civil unrest with an isolationist anti-intellectual theme. This may not be an extinction event (for humans anyway), but it won't be a lot of fun. 
  
  **Global Thermonuclear War / The Fallout Premise**   
  A world ending nuclear war seems far fetched right now, but it would probably only take a generation or so of a government evolving in the wrong direction before it might be willing to threaten its resource rivals (after all, that's all it took to go from the cold war to current global politics). Actually, Putin would only have to be a little bit more insane for us to be there already.
  
  **Runaway Climate Change**   
  I haven't seen a lot of evidence that runaway climate change can be triggered by a 2C rise, but who thinks that global warming will be restricted to that in the following decades and centuries? If the most pessimistic of climate scientists are right we could end up Venusing the Earth over the next few centuries. I'm not urgently afraid of this, but we're getting into less likely territory.
  
  **Solar Flare**   
  A big solar flare could knock out unshielded technology on earth. Humanity would probably survive, but I'm not sure civilization could. This could probably be inserted into the list more accurately by looking at how often they happen compared to asteroid impacts, but this is in fear order, not probability order.
  
  **Asteroid Impact**   
  An oldie, but a goodie. Not particularly exciting or glamorous, but we know it happens, it's happened before, and even if we see the asteroid with humanity's name on it years in advance there's no guarentee we'd be able to generate the technology (or even, a cracked and jaded part of me thinks, the political will) to tractor it off course.
  
  **Cronus Apocalypse**   
  I'm lumping all of the extinction events where one of humanity's technological children kills its creator, either by bug or malice. Terminator, Matrix, Grey Goo and Paperclip all own less of my fear space than even an asteroid impact - not only would we have to make a lot of mistakes to get there, but we're going into it with our eyes so wide open. There are already a lot of smart people thinking about how to avoid this.
  
  **Exotic Physics**   
  These are fun ones. We either accidentally generate strange matter or are hit by a roaming quark star, and the whole Earth is converted to strangelets over a period of time. Maybe strangelet production in colliders is the great filter? Fun. Also - a vacuum metastability event. Not necessarily possible or likely, but the untimely end of the *entire universe* has to get an honorable mention.
  

  • u/None:

    I basically agree with your whole list, but there's one thing to note: the Apocalypse Level of the event.  FHI, for instance, has a tendency to, in my personal opinion, underrate "Econ-Eco Apocalypse" merely on grounds that it would not actually *completely annihilate humanity or technological civilization*.  I consider this a problematic assessment, because I think there *is* a chance it would destroy technological civilization, setting us back to a permanent low-productivity, low-energy, nigh-Malthusian existence.
    
    > I'm lumping all of the extinction events where one of humanity's technological children kills its creator, either by bug or malice. Terminator, Matrix, Grey Goo and Paperclip all own less of my fear space than even an asteroid impact - not only would we have to make a lot of mistakes to get there, but we're going into it with our eyes so wide open. There are already a lot of smart people thinking about how to avoid this.
    
    The big thing that reduces my worry about UFAI is that even a recursively self-improved superintelligence does have sample complexity and computational complexity bounds it *cannot* exceed, and the *first* AGI agents *will not be* recursively self-improved superintelligences.  They won't be *able* to self-improve without first gathering enough data and performing enough processing on it (call it "education") to form an accurate, naturalistic model of the world that includes itself and includes the necessary understanding to code an improved agent.
    
    *That* phase will take time and data, lots of it, during which we humans will still have the advantage and *probably* be able, if the makers have *bothered* to take decent precautions in the first place (see: entire sub-field of Corrigibility, currently in its infancy), to shut the damn thing off, by force if necessary.
    
    Which isn't to say there's no risk.  It's to say that the risk is more on the order of massive radiation spills than on the "INSTA-KILL" level.
    

    • u/eaglejarl:

      One additional thing that worries me about civilization collapse: we've used up a lot of the easily available fossil fuel and other resources. Going from our current state to "unlimited" energy (fusion, lots of fission, space based solar) is quite doable.  If we had to start technology over from scratch, would we be able to get across the leap from steam to any of those methods, or have we / will we have used up the intermediate energy sources?
      

      • u/None:

        And that is *exactly* what worries me when I think about these issues.  Fuck, at the moment I just wish we had the balls as a species and a geopolitical world to actually build out nuclear energy, both for decarbonization *and* for energy-scarcity reasons *and* to give us a firm foundation for developing more advanced technologies without the fear of resource collapse.
        

        • u/eaglejarl:

          I read an interesting white paper on thorium reactors. If true, it would be the holy grail of energy supply.
          

          • u/None:

            [Sweet Cthulhu, there *was* a functioning prototype running safely and efficiently at net energy gain!](http://en.wikipedia.org/wiki/Thorium-based_nuclear_power#Background_and_brief_history)  And those ABSOLUTE FUCKTARDS abandoned the line of research because it was *too nonviolent!*  God fucking damnit!
            

      • u/iemfi:

        But there's still plenty of coal. Also hydro, wind, and geothermal seem easy enough to access with limited technology. And retaining and passing on knowledge seems like it would be a priority for survivors.
        

        • u/eaglejarl:

          Yep, agreed on all points.  If we were to collapse *now*, I would have no fear of our ability to restart.  But what if we collapse in a century or two, when the easily accessible coal has been exhausted?   Hydro and wind are only useful in certain places...
          
          [google]
          
          Hm, actually, hydro seems to have pretty good energy supply -- a quick search says that China produced 721 Tw/hr in 2010.  Maybe that *would* be enough to restart.
          
          Cool, thanks.
          

      • u/azripah:

        Energy wouldn't be too bad, really. Solar power generators date to the 19^th century and don't really require any materials that couldn't be obtained with... traditional, let's say, methods.
        
        http://upload.wikimedia.org/wikipedia/commons/6/66/Mouchot1878x.jpg
        
        It's the materials that are derived from petrochemicals that would be annoying to do without. Without even going in to all the materials surrounding you made from plastic, think of all the fertilizer and people supported by that fertilizer that is currently derived from fossil fuel resources. Feeding everyone currently on the planet while lacking the benefits of the green revolution would be taxing to say the least.
        

  • u/Topher876:

    I'm not sure if it's what scares me the most but the thing that makes me most angry is a the possibility of an anti-intellectual  movement taking over. There are a lot of ways Society could fall apart but to think we could make a conscious decision to abandon our progress just makes me shudder
    

• u/DangerouslyUnstable:

  Asteroid strike. We don't do enough monitoring and haven't put enough money into doing something even if we see it. Nukes won't work, especially if we see it late which we probably will.
  

• u/iemfi:

  Well, I'll be the first to say AI by a big margin. The rest either won't result in extinction, are incredibly unlikely, or both. They also have the advantage of being obviously bad. Like if we detected an asteroid on a collision course we wouldn't be worrying about whether it was harmful, we would throw everything at it. We've also had the tech to wreck the Earth for some time and haven't done so. So I don't see why nanotech, super bugs etc, would change that.
  

• u/Oh_Hi_Mark_:

  Simulation shutdown. There are probably things we could do to cause or avert it, and we'll never know what they were until it's too late.
  

• u/Nepene:

  A solar flare. It would utterly disrupt everything electronic, happens on a regular basis, and is too expensive to cheaply deal with. 
  
  An asteroid strike we can cheaply deal with with a gigaton nuke. We program AIs and nanotech, we're probably going to program safeguards.
  

  • u/eaglejarl:

    I'm honestly not sure: are we able to build a gigaton nuke? Largest ever was only 50 megatons. 
    
    Even if we can, how would we get it there? We currently have nothing capable of launching to beyond LEO. We do still have the plans for a Saturn V -- I think; I've read that they were lost -- but could we build, test, launch, and have it arrive in time?
    
    If the asteroid came from the inner system (e.g. an Apollo), we quite possibly wouldn't see it until it hit us, as the sun would be behind it. Even if it comes from the outer system and were spotted with fifty or a hundred years on the clock, I would be worried about people procrastinating and / or arguing about who should put in how much funding etc that we never actually deal with it in time. 
    
    Nuking the asteroid head-on would likely not help -- at most it would break it into smaller chunks with the same mass and they would still hit us. If they were small enough then *maybe* they would all burn up, but I question the ability of mankind to build and deploy a weapon capable of vaporizing or thoroughly decomposing a 10km chunk of nickel-iron. 
    
    If we hit it at an angle then maybe we could deflect it enough that it would miss us, but that would rely on having detected it early enough, which isn't a certainty. 
    
    What safeguards would we build on a fast-takeoff AI that would ensure our safety?
    
    What safeguards would we build on nanotech that would ensure our safety?
    
    Assuming such safeguards exist, once nanotechnology / AI is real, it will become cheap enough to be accessible to construction by small groups and corporations. With enough labs building it, someone will be careless or crazy or stupid or evil enough not to incorporate the safeguards.
    

    • u/Nepene:

      http://en.wikipedia.org/wiki/Asteroid_impact_avoidance
      
      >Following the 1994 Shoemaker-levy 9 comet impacts with Jupiter, Edward Teller proposed to a collective of U.S. and Russian ex-Cold War weapons designers in a 1995 planetary defense workshop meeting at Lawrence Livermore National Laboratory (LLNL), that they collaborate to design a 1 gigaton nuclear explosive device, which would be equivalent to the kinetic energy of a 1 km diameter asteroid. This 1 Gt device would weigh about 25-30 tons being light enough to be lifted on the Energia rocket and it could be used to instantaneously vaporize a 1 km asteroid, divert the paths of extinction event class asteroids (greater than 10 km in diameter) within a few months of short notice, while with 1 year notice, at an interception location no closer than Jupiter, it would also be capable of dealing with the even rarer short period comets which can come out of the Kuiper belt and transit past Earth orbit within 2 years. 
      
      You can basically scale up nuclear bombs as much as you like, they just add more weight and are less effective than many smaller bombs at killing cities. As noted, you can do this with far less warning than for most weapons. You'd probably use existing crafts as much as possible, maybe build extra engines if necessary.
      
      All the heat you dump into an asteroid vaporizes the material and is ejected. It causes the asteroid to fly off in a different direction.
      
      >What safeguards would we build on a fast-takeoff AI that would ensure our safety?
      
      You'd probably program them with a certain set of ethics and desires that excluded mass death of humans, as appropriate for the application.
      
      >What safeguards would we build on nanotech that would ensure our safety?
      
      A kill switch, need for some rare resources to grow, variants of that.
      
      >Assuming such safeguards exist, once nanotechnology / AI is real, it will become cheap enough to be accessible to construction by small groups and corporations. With enough labs building it, someone will be careless or crazy or stupid or evil enough not to incorporate the safeguards. 
      
      For the most part I'd imagine people would use standardized ones like microsoft or linux or apple, and ones that would resist any insane AIs. 
      
      For nanotech, it's fairly cheap to kill it, not as big an issue. Flamethrower kills pretty much any nanotech, emp.
      

• u/ianyboo:

  I worry about the small things like the economic upset from Walmart (or whoever) automating its workforce, other companies having to do the same and then massive unemployment, Ferguson style riots, a complete loss of control in america and the rest of the world collapsing with us.
  
  Sort of a Elysium style apocalypse. Not necessarily an extinction event, but enough of a slowdown that we remain a single planet species for an extra hundred years and catch a rogue asteroid for our stupidity.
  

• u/Chronophilia:

  Nuclear war, certainly.
  
  It seems like all cataclysms have a sweet spot of technological development where they could wipe out the human species. Well, perhaps a "bitter spot" is a better name for it.
  
  Earthquakes or disease are too late - we've been dealing with those for our civilisation's entire history, and we've got good enough infrastructure to handle everything that's been thrown at us so far. Diseases aren't getting more advanced, but healthcare is. Arguably, a species-killing epidemic would have an easier time spreading across nations and continents, since we've got aeroplanes now... but I think it's safe to say that we're better equipped to deal with pandemics than we ever were before.
  
  Nanotechnology and AI are too early - the technology to create them simply doesn't exist, and I find it unlikely that anyone working on any top-secret project has beaten the cutting edge by enough to change that.
  
  Nukes work, they're here now, and the only reason we haven't already been wiped out is that the people with the keys are under a lot of pressure to not use them. If the apocalypse arrived tomorrow, I'd wager that it would be a nuclear one.
  

  • u/AmeteurOpinions:

    > Diseases aren't getting more advanced
    
    Read up on the [ misuse of antibiotics](http://en.m.wikipedia.org/wiki/Antibiotic_misuse). We're slowly making strains of bacteria immune to the most common and affordable treatments, which will be a rather expensive problem within our lifetimes.
    

    • u/Chronophilia:

      Oh yeah, I forgot about that. Still, my point stands - we've only had antibiotics since 1928, so anything that hasn't killed us in the last million years is unlikely to have become an existential threat in the last 84.
      

      • u/notmy2ndopinion:

        I disagree -- to put it colloquially ala Ian Malcolm: "Life finds a way."
        
        You are dismissing illness as the MAJOR THREAT because we've invented some medications?  Antibiotics, pesticides and any other wiggly-killer will become outdated as high-replication rates put a selective pressure seeking novel mutations for survival.
        
        We are guiding the hand of evolution.
        
        Frankly, an existential threat I worry about is the next terraforming microorganism that becomes ubiquitous.  Precursors to chloroplasts flooded Earth with potent oxidants and poisoned the air, causing massive extinction with OXYGEN.  Who know what sort of changes could terraform our planet in terrible ways that we can't escape, even if we come up with gravity equations to float to Saturn?  (think Blight from Interstellar, except it is infectious and removes the only source of sustenance and air for a species of consumers, reliant on OTHER creatures for energy.
        

• u/ajuc:

  Biotechnology developing so well, that people will have bioprinters on their desktops. Someone inevitably will produce plague to end all plagues and that will be it.
  

• u/Vermora:

  Deranged, determined individuals with a lot of power.
  
  It is currently possible for one person, working independently and alone, to build or purchase a weapon (perhaps a bomb or a gun) and use it to kill a few people, if they are intelligent and determined enough.
  
  As technology develops, the amount of raw physical damage that a person is capable of dealing increases, while humans are not generally getting physically tougher.
  
  It is conceivable (although not necessarily feasible) that in the far future, science and technology will advance to the point that a small group of highly deranged, capable individuals would be capable of dealing catastrophic damage to the human race as a whole.
  

  • u/FaceDeer:

    Important to consider in this scenario, though, is the fact that increased technology also allows for people to build new defenses against these sorts of things. If biotech reaches the point where a deranged individual can download a plague kit off of the internet and mix up a horrifying new pandemic, that same technology allows everyone else to download a vaccination kit to neutralize it.
    

---