5.4 KiB
[EDU] "How super AI could end the age of humans", with Nick Bostrom. A pretty good overview of the dangers inherent in intelligent agents with their own goals (35:04)
-
Author: u/Pluvialis Second Age Sauron*
-
Score: 10
-
Created: 2014-08-10T18:23:01
Post:
Comments:
u/Traiden04 [+1] (a day later)
So I have been talking with friends that I game with and have asked thier opinion on this topic a few times. They are unconvinced of the potential threat an AI optimizer could cause, stating that there is a lot of hand waving going on in the fact of an AI being unable to change its own goals so that they might only just optimize themselves instead of perusing the noble goal of converting all matter into paperclips. I have been unsuccessful at reasoning with them about how an AI is not the same as human intelligence and that it would not see the benefit in changing its own goals.
Edit: Further clarification on the subject brings forth the question on wether an AI could independently question its own goals and values and change them.
u/Pluvialis [+1] Second Age Sauron (a day later)
The comments in /r/skeptic where this topic is also being discussed really demonstrate the need for a popular movie or two in which an AI escapes its box.
u/None [+3] (3 days later)
Oh my dear fucking God, they're actually denying the orthogonality thesis.
u/Prezombie [+1] (3 days later)
For some reason, I see strong AI as not as a threat, but simply a possible next step.
Evolutionarily speaking, the two main ways a species goes extinct are an invasive species dominating the niche of the native one, or that native species having enough beneficial mutations that spread through the populace until none of the original flavor are left in that biome.
In that sense, many of these people are equating strong AI with the former kind, but to me, I see strong AI as the latter kind.
I wonder if this is just me having an abnormally weak bias for DNA-based descendancy.
u/None [+4] (3 days later)
The "some reason" is that you're conceptualizing the universe through a neo-Darwinian meta-narrative, rather than chucking out your metanarratives entirely and assigning moral valences where you second-order want them.
u/Pluvialis [+1] Second Age Sauron (3 days later)
Well, I think I understand the sentiment of our AI 'child' sort of 'inheriting' the universe from us, but how would you feel about an AI that killed us all and then went around doing something totally worthless, just because we programmed it badly? Like going around turning every source of energy into computational resources for itself until in short order it has absorbed the universe and sits just running cycles on loop searching for further threats against itself or something equally mundane?
u/Prezombie [+1] (3 days later)
Yeah, that would suck. But a strong AI wouldn't be built in a vacuum. Sure, one strong AI could be a mundane paranoid paperclipper, but there's also human children who grow up to be criminals. Due to social constraints, criminals rarely thrive in a complicated environment if the entire social structure is against them.
Similarly, a single strong AI would be dangerous, but one paranoid paperclipper wouldn't be able to take off if there's hundreds or thousands of Strong AI in a social network of checks and balances.
u/jalanb [+2] (6 days later)
but one paranoid paperclipper wouldn't be able to take off if there's hundreds or thousands of Strong AI in a social network of checks and balances
Humans did, why not (a particular species within) machines?
u/CaesarNaples2 [+1] (20 hours later)
Hi, thanks for the link.
SPOILERS:
The two goals for researchers are basically (at the end):
creating AI
controlling AI
Goal 2 should be reached before goal 1. Basically. Yet, massive economic pressure to just create AI is hugely outclassing the effort to control it.
/spoilers
u/None [+2] (3 days later)
Goal 2 should be reached before goal 1. Basically. Yet, massive economic pressure to just create AI is hugely outclassing the effort to control it.
Really? Because I don't actually see that much economic effort being poured into AGI, compared to how much goes into most other fields of theoretical computer science.
What I will say is that people mostly don't seem to work on Friendly utility functions for several reasons:
They think they lack the philosophical framework to conceptualize a "controlled" or "safe" utility function besides reinforcement learning.
They think they lack the mathematical ability to describe a non-learned utility function at all (this is true: we do currently lack that mathematical ability).
They think reinforcement learning will be good enough, since after all it's been ok up to now.
The fact that they don't just endorse doing whatever their dopaminergic circuits consider the Most Interesting Thing at any given time doesn't seem to occur to them.