58 KiB
[D] Monday General Rationality Thread
-
Author: u/AutoModerator *
-
URL: https://www.reddit.com/r/rational/comments/3xpjv9/d_monday_general_rationality_thread/
-
Score: 27
-
Created: 2015-12-21T15:04:59
Post:
Welcome to the Monday thread on general rationality topics! Do you really want to talk about something non-fictional, related to the real world? Have you:
- Seen something interesting on /r/science?
- Found a new way to get your shit even-more together?
- Figured out how to become immortal?
- Constructed artificial general intelligence?
- Read a neat nonfiction book?
- Munchkined your way into total control of your D&D campaign?
Comments:
u/Vebeltast [+16] You should have expected the bayesian inquisition! (8 hours later)
Does anybody know why Spacebattles and Sufficient Velocity hate the Rationality meme-system? I haven't been able to get an answer out of any of them other than "Yudkowsky's navel-gazing cultish nonsense", much less a reasoned dissenting argument that'd I'd be able to update on. Did Methods of Rationality kill all their pets or something?
u/blazinghand [+16] Chaos Undivided (10 hours later)
Rationality in general has a PR problem. People hear about it and based on whatever past experiences, dismiss it right away. Individual tenets of rationality, or even the whole hog, are accepted by people if you don't introduce them as rationality. You can put lipstick on this pig.
Of my friends, some hate the rationalism, and the one who hates rationalism the most is also the one who uses it the most. It's just a name / branding issue really. Stuff like the ideas in Beware Trivial Inconveniences or The Toxoplasma of Rage or whatever rationalist article, if presented without rationalism mentioned, are usually really popular. I can just take the idea, present it myself, and people will like it. It's hard to give them follow-up reading though.
It's just a bad brand. I can't speak about SB and SV specifically, but that's just what I've observed.
u/alexanderwales [+13] Time flies like an arrow (13 hours later)
I remember talking to some people on LessWrong a few years ago about why the brand was a bad one and getting some combination of denial ("It's not a bad brand!"), obstinate refusal to see this as a legitimate problem ("It's a bad brand because we say things that are true!"), or placing blame on others ("It's the haters!"). It just convinced me that I wasn't likely to have a productive conversation on the matter. Same with the "cult" stuff, which is closely related.
u/None [+12] (13 hours later)
My own pet peeve on that score: why is "the Sequences" usually (or often) capitalized?
For purposes of comparison, Christians like to capitalize "Old Testament" and "New Testament," "the Koran" is capitalized, etc.
It's not a big deal, and I suppose most people don't pay much attention to details like that -- but I've always found it a little creepy.
u/Vebeltast [+8] You should have expected the bayesian inquisition! (14 hours later)
I've observed a couple people as they read through the sequences. I think that it's capitalized like the other books because it has comparable power. If it's new to you, you can understand, and you it buy into it, you can build most of your personal philosophy around it. It's about as creepy as, say, an Asian person reading a Bible translation around the age of 20 and suddenly becoming a furious Christian.
u/None [+7] (a day later)
If it was all placed in the context of traditional academic statistics and philosophy, it would seem a fair bit more commonsensical but a fair bit less Deeply Profound.
Ironically, the thing I like most about this subculture is that we value the commonsensical and the natural over the Deeply Profound.
It was Eliezer who said they're 85% non-original material, even though they don't cite much.
We need way better introductory books.
u/Vebeltast [+4] You should have expected the bayesian inquisition! (a day later)
Ironically, the thing I like most about this subculture is that we value the commonsensical and the natural over the Deeply Profound.
See, that's the interesting thing that I've just realized: a lot of what we see as being common-sense is, to people who haven't seen it before, Deeply Profound. Like, the bit about death being bad: We see it as blindingly obvious. Death is simply a bad thing and I can't see anything in the laws of physics that demands it, and so I would prefer the universe where nobody has to die. But if you say all that to someone who hasn't thought about it, you end up Deeply Wise!
Agreed on needing better introductory material. EY's single biggest achievement is that he put all of this in a single place and organized it so a single person can assemble it for themselves. It's an important achievement, but it can be duplicated more easily now that it's been done once. We just have to get it put in more places.
u/None [+2] (a day later)
See, that's the interesting thing that I've just realized: a lot of what we see as being common-sense is, to people who haven't seen it before, Deeply Profound. Like, the bit about death being bad: We see it as blindingly obvious. Death is simply a bad thing and I can't see anything in the laws of physics that demands it, and so I would prefer the universe where nobody has to die. But if you say all that to someone who hasn't thought about it, you end up Deeply Wise!
Oh right. As a group, we're split into the five-year-old children and the meta-contrarians. Oy.
(Although the Second Law of Thermodynamics does seem to demand a heat-death of the universe eventually. That's just not relevant to our timescales right now, unless you're searching for Eternal Deep Truths to solidify a worldview.)
u/Vicioustiger [+2] Just trying to be a better person. (14 hours later)
I don't know what "the Sequences" are, but doesn't the "the" imply it is a name, making it a proper noun? Therefore always capitalized.
u/None [+3] (14 hours later)
https://wiki.lesswrong.com/wiki/Sequences
I'm not sure whether it is a name (the webpage I linked to above is titled "Sequences" but has both EY's collection of posts that are usually referred to this way with a capital S, as well as other sequences by different lesswrong-affiliated authors).
Regardless of whether it is a name, I still find it a little creepy to see someone told to "read the Sequences."
Just something off about that. Although: I may be the one off here, I suppose creepiness is in the eye of the beholder.
u/Vicioustiger [+3] Just trying to be a better person. (14 hours later)
I can understand that worry and comparison after reading some of the other comments here. The word cult has been used at least 5 times just discussing it, and when the entire point is to come to rational conclusion then anything with a cult connotation would seem off-putting.
u/blazinghand [+2] Chaos Undivided (a day later)
I always assumed it was because Yudkowsky was planning on turning them into a book or something. Prolegomena to Any Future Metaphysics is capitalized because it's a title, even if it's a purely descriptive title.
u/None [+3] (a day later)
You may be correct (and I believe he did turn them into a book). Still, even so, "read the Sequences" sounds exponentially more creepy than "read Plato's Republic," no?
u/Rhamni [+3] Aspiring author (a day later)
But in all seriousness, do read Plato's Republic. With footnotes.
u/blazinghand [+3] Chaos Undivided (a day later)
I haven't actually run into anyone who's told me either of those things in response to a query so I can't say in context. Comparing "Read Yudkowsky's The Sequences" vs "Read Plato's The Republic", the latter sounds better, but this to me again boils down to a branding issue. If I wrote a book called Modern Cognitive Science and You: Seventeen Easy Steps to Success, even if it contained the same content, you'd have a real different experience recommending it to people. Same if a famous cognitive scientist wrote it and gave it a more professional title.
I'm sure it's not helped by rationalists suggesting it in a strange way, either. People in general don't know how to sell things. I doubt rationalists are an exception.
u/aintso [+1] (2 days later)
Wait, how is that that people don't know how to sell things? I though people being social creatures and being capable of empathy implied that they had some capacity for manipulation. This is really trivial but looks like I had it wrong the whole time. Thank you for pointing this out.
u/alexanderwales [+3] Time flies like an arrow (2 days later)
Selling things is hard. In order to be capable of making a sale, you need to be able to compete with millions of other, better sales agents out there. If you're not able to do that (which mostly people can't, not with just standard social manipulation and empathy) then you're not able to make a sale, and thus don't actually know how to sell things.
I think sales is a really common thing to get Dunning-Krugered on, since I've seen a lot of really inept people trying to sell things (including rationality).
u/None [+2] (a day later)
Still, even so, "read the Sequences" sounds exponentially more creepy than "read Plato's Republic," no?
I think that depends on whether you know the actual content of Plato's Republic.
u/Rhamni [+2] Aspiring author (a day later)
I mean, the eugenics stuff isn't even well run. A yearly rigged lottery? You don't think people will end up having sex outside of that?
In all seriousness, he was a very thoughtful, intelligent man who lived in a society that thought slavery was ok and became the cultural capitol of 'Greece' by using money raised as tribute. Fortunately the main message is not that you should agree with him on every point. It's that you should collaborate with others, analyse arguments thoroughly, discard the ones that don't hold up, even if they come from him, and keep searching honestly for the truth.
u/None [+2] (a day later)
It's that you should collaborate with others, analyse arguments thoroughly, discard the ones that don't hold up, even if they come from him, and keep searching honestly for the truth.
And also that slave-taking is fine, virtue-ethics is a thing, all objects are mere projections of perfect Forms that live in a Heaven of Ideas, etc.
Frankly, I'm not willing to let any one thinker or group of thinkers claim ownership over basic critical thinking, in the same way that they don't get to "own" physics.
u/Rhamni [+2] Aspiring author (a day later)
Fair enough, although I will point out that in Plato's imagined Republic, there are no slaves. There is a caste system, but all the material wealth stays at the bottom, while political power comes with forced asceticism and gender egalitarianism. Children are assigned caste independently of their parents, depending on how well they do in school (although the eugenics program suggests he expects most apples to fall near the tree). It's clearly far from a society I or others of today would endorse, but while the realm of the forms and all that jazz is plainly silly, the critical thinking was presented in a way that helped me become more interested in philosophy. Obviously Plato does not 'own' critical thinking, but he's an early master of it.
He is not in any way mandatory reading, but he was an excellent starting point for me, and I still enjoy reading a dialogue every now and then.
u/Uncaffeinated [+11] (12 hours later)
The problem with talking about "rationalism" like this is that you seem to be conflating multiple ideas. It's like, rationalism is about making smart choices, which noone can argue against, and then oh by the way, you're supposed to believe in evil AIs going FOOM and donate to Yudkowski now.
u/blazinghand [+13] Chaos Undivided (12 hours later)
Good point! That's what I'm talking about.
There's definitely some terminology problems here. "Rationalism" as it is used refers to a bunch of different ideas, some of which people like, and some of which people do not. This is exactly why, when you want to talk the things you want to share, you don't call it rationalism.
In a similar vein, when I try to introduce other things (like socialism or libertarianism or whatever charged idea there is) I don't call them by name. Names and labels hurt people's ability to be good about this kind of thing.
u/None [+3] (a day later)
Also, "rationalism" means Descartes and "rational" has a tendency to be used as "Think what I tell you to!"
u/MrCogmor [+10] (9 hours later)
I'd guess because they have had a number of obnoxious posters trying to encourage people to read less wrong or support the singularity institute. Probably also bits of appealing to Yudkowsky as an authoritative source despite his lacking of credentials the opponent would find meaningful.
There was/is a personality cult around Elizier because of the halo effect and people creating an image of him as a person based only on his high rated and carefully crafted posts.
While lesswrong has a number of transhumanist memes, the ones that are only really associated with lesswrong tend to be the weird and implausible ones like the A.I foom theory, worry over existential risk, roko's basilisk, cryonics and overuse of the word rationalist as an adjective.
The community does resemble a religion in a number of respects. The meetups, solstice celebrations, the insular community, the weird beliefs and most importantly the appeals for money from the machine intelligence research institute (formerly known as the singularity institute)
People do stereotype members of the rationalist community. They are wrong to write the community off as a whole but they do have a point. The people that do deliberately advertise that they belong in this community tend to be the obnoxious posters I mentioned earlier.
u/None [+5] (21 hours later)
[deleted]
u/Vebeltast [+8] You should have expected the bayesian inquisition! (23 hours later)
...Last time I saw them, maybe a month ago? Tim Josling was in the process of doing that when I poked the Less Wrong slack chat to see if it was interesting, testing how hard it was for random people on the subway to pick up some of the simpler ideas against bias. Maybe we're just hanging out with different rationalists?
u/traverseda [+3] With dread but cautious optimism (a day later)
Last week my roommate was very pissed off at how hard it would be to run an independent study on bacopa monnieri, compared to quick and dirty trials we can run on other drugs.
Yesterday I tried to make coloured-flame candles and used basic science throughout. I used basic science to figure out seam strength when bonding two pieces of mylar space blanket together just last week.
We constantly use science to binary search though our 3D printers' problem space.
Science is not what lesswrong brings to the table though. It's impossible to do any kind of engineering job without at least a basic adherence to the scientific method.
A lot of the rationality techniques that I value most aren't just basic science though. When I did a CFAR workshop that was something that kept coming up, the cost of information and dealing with uncertainty.
As an individual, you don't have the time or resources to test your questions against reality.
Take, as an example, the question of what career to take, or which job offer to take. The scientific method won't help you here.
People conflait lesswrong style rationality with science because
theywe talk about science a lot. But science is only one tool in the toolkit, and although it's often useful in my day to day life, it's only useful when your claims are testable.The practical explanations of cognitive biases, cached thoughts, etc are really what make it a useful toolkit.
u/SvalbardCaretaker [+1] Mouse Army (a day later)
Why did you need to bond two space blankets?
u/traverseda [+2] With dread but cautious optimism (a day later)
I want to build a space blanket tent, like the double walled inflatables they use near the arctic.
I like a lot of the libertarian ideas around start up cities and seasteading, but using conventional construction the startup costs are just too high. I'd like inexpensive open source infrastructure to be a thing.
This is an early test in patterning mylar to make a sort of bubble-wrap type surface. About soft-ball sized bubbles.
One of my other goals, well more of an aesthetic then a goal, is self-sufficiency and ultra portability. This potentially tackles that pretty well as well.
I also have most of a yurt. It's a lot less portable then my ideal, and probably unreasonably expensive to heat then the mylar if it works well.
u/SvalbardCaretaker [+1] Mouse Army (a day later)
I think inexpensive land construction is pretty much a solved problem now that house-printers are out, so you can focus on solving "portable shelter"!
Sounds nice.
u/ayrvin [+2] (2 days later)
Are house printers actually cheaper than building a house the normal way?
u/SvalbardCaretaker [+1] Mouse Army (2 days later)
not yet. Technology not yet fully developed, low hanging fruit not yet picked, no economy of scale etc. However, unlike 3D printing complex objects, where economy of scale and build quality really favour a centralized approach (despite what 3d printing enthusiasts say) house printing does not have the same issues.
However, seeing that labor costs make about 50% of house projects the advantages should be obvious. Naturally not all or even most of that is going to be masonry, (maybe 20?) but you should be able to have reduced build times as well. House owners and investors should like hugely reduced build times.
The construction savings are most evident if the houses you build dont need a lot of afterwork finish, eg. basic houses, shelter for the very poor or refugees. Without tons of plumbing, internal wiring, a basement etc. This is where we expect to see first widespread deployment, and indeed that is where we see the stories currently breaking taking place.
TDLR: Extremely basic houses, yes. Modern amenietes(?), developed world houses, not so much, and not until a while.
u/traverseda [+1] With dread but cautious optimism (2 days later)
House printers have a bunch of problems. Cement isn't that good of a material, for one thing. Another is that they have serious problems with overhangs.
u/None [+6] (8 hours later)
What is the rationality meme-system?
u/SvalbardCaretaker [+14] Mouse Army (9 hours later)
The memplex of Xrisk,AI-Xrisk, effective altruism, human biases,bayesian calculation, and evopsych that originated on lesswrong.com. Eg. Harry Potter James Evans Verres style thinking.
u/Vebeltast [+8] You should have expected the bayesian inquisition! (9 hours later)
Basically anything related to Rationality as used here, utilitarianism, friendly AI, existential risk, etcetera. I've seen hostile reactions to the mere mention of biases, black swans, or recursive self improvement.I can't figure out why, either, because the reaction is never explained past "it's nonsense".
u/Nighzmarquls [+11] (10 hours later)
Considering how much sufficient velocity and space battles pushes for starships that make very little sense, I'm pretty sure the "it's Nonsense" thing is not the real answer. But they as a total group might now know themselves.
Incidentally I've started crossposting a story that is threaded with a whole bunch of rationality stuff to their forums and responses are pretty positive so I think it's likely that their actually more concerned with the 'dressings' of rationality being distasteful and not the actual core ideas.
u/None [+8] (12 hours later)
[deleted]
u/Rhamni [+3] Aspiring author (21 hours later)
While Eliezer's first post yelling at Roko was a very unfortunate, and the least calm thing I've ever seen him write, I think it's painfully clear that it's a non-issue E. and all of Less Wrong have no interest in talking about, but others like to bring it up again and again because it sounds silly, especially if you haven't read anything about all the stuff you need to read about for the idea to make sense.
I have tried to give Less Wrong a chance a few times, but it doesn't capture my attention. I read a few sequences every now and then, I like this sub, and... well, that's about it.
u/Vebeltast [+2] You should have expected the bayesian inquisition! (23 hours later)
It was reading about Yudkowsky, MIRI, and the reaction to Roko's basilisk.
I guess that's sort of what I'm wondering about. Spacebattles et al. seem to be completely on board with 95% of the individual ideas if presented on their own - all you really have to do is rephrase them and post them in isolation - but if you mention that you got it from LW it's suddenly rejected. As if the argument's validity is somehow dependent on who came up with it first. "They're rejecting what they see as a cult" might explain that, though.
u/None [+5] (a day later)
People don't like to be fed "hooks" where you start with seemingly commonsensical ideas and end up with radical, implausible-sounding stuff, at least not as Author Tract-y stories. If you think your logic is airtight, you'll usually just talk to scholars. If you want to write a story, you keep the weird-logic in the background and let people work things out on their own.
People have very little reason to shift their fundamental, semi-metaphysical beliefs about the world just because someone's preaching at them. In fact, it's rude and gets people mad.
u/None [+3] (a day later)
"Black swans" are indeed a load of bullshit. If your model (eg: Black-Scholes Equation) puts an extraordinarily low probability on an event (eg: demand-starved, debt-driven financial crisis) that other models (eg: conventional Keynesianism) called practically inevitable, and which has happened before (Great Depression), it's just a bad model.
u/Vebeltast [+1] You should have expected the bayesian inquisition! (a day later)
...Yes? AFAIK "black swans" are just another manifestation of optimism bias. If you can deal with your optimism bias directly by saying "my model is probably not as right as I think it is and I should prepare fallbacks for if it turns out to be an awful model", great! If not - and this is probably the case more often than not - here's another tool you can use to formalize (and therefore regress-toward-mean the success rate of) the process of removing optimism bias.
u/None [+1] (a day later)
See, I was under the impression that Nicholas Taleb had introduced this weirdo idea of "Black Swan" events not as flaws in your model, but instead as innately unpredictable events which no reasonable model could hope to capture, but which nevertheless occur frequently enough that we all need to make "antifragile" policies for responding to them.
u/Vebeltast [+2] You should have expected the bayesian inquisition! (a day later)
Hmm, that might be the disagreement. To me, "there exist innately unpredictable events" means "there is a theoretical cap on the accuracy of any model". Which, maybe? Map-territory distinctions and computational complexity theory do suggest to me that there are systems which can produce events that couldn't have been predicted by any model simpler than the system itself.
u/Uncaffeinated [+4] (12 hours later)
Well I can't speak for them, but I can say why I don't like it.
At its worse, the community seems more like a cult than a group of people interested in overcoming biases and well thought out fiction.
For example, Friendly AI/Singularity stuff is just Rapture without the Jesus, AI-X Risk is Caveman Scifi for the modern age, Roko's Basilisk is Pascal's Wager with the serial numbers filed off (though at least noone takes that seriously) etc.
For all its focus on being rational, there's a lot of outlandish ideas passed around without any critical thinking.
u/None [+4] (a day later)
And this is why our cult leader's most under-appreciated saying is, "Beware things that are fun to argue about."
u/Vebeltast [+2] You should have expected the bayesian inquisition! (23 hours later)
any critical thinking
Perhaps the critical thinking is there you just haven't seen it being done? For example, it sounds like you're conflating at least two of the different versions of the singularity. I mean, a recursive self-improvement explosion is clearly a thing that could actually happen - we could do it ourselves pretty trivially if we didn't have all these hangups about medical research with psychedelics or if we dumped a spacex-sized pile of money into brain-computer interfaces - and the risk of unfriendly AI is obvious enough that Hollywood has been making movies about it since the 60's, though as always the real deal would be much more subtle and horrifying. I'll give you the initial response to the Basilisk, though; it's a non-issue now that people have realized that it's a wager and deployed the general-purpose wager countermeasure, but the flawed memetic form is still floating around causing problems.
I can see how it would be extremely cultish if viewed from the outside, though. It's a large, obviously coherent system of beliefs, with a consistent core and an unusual but relevant and deep-sounding response for many situations, and that gives it the seemings and feelings of deepness that you usually only see in religions. And then it comes down to whether your first impression suggests "Bible" or "Dianetics".
Probably explains why 95% of it is well-received if delivered on its own. Without the rest of the large mass giving it unusual coherence and consistency, it seems like just an awesome idea rather than a cult. Which would kind of explain the success I've had directing unsuspecting people to just the sequences, since by the time they've gotten to critical mass they've bought into most of what they've read.
u/Uncaffeinated [+5] (a day later)
I suppose this is a side tangent, but I'm fairly skeptical about the scope for recursive self improvement.
First off, it's hard to make an argument that doesn't already apply to human history. Education will make people smarter, and then they figure out better method of education and so on. Technology makes people more effective and then they invent better technology, etc. Humans have been improving themselves for centuries, and the pace of technological advance has obviously increased, but there's no sign of a hyperbolic takeoff, and I don't think there ever will be.
The other issue is that it flies in the face of all evidence and theory. Theoretical Computer Science gives us a lot of examples where there are hard limits on self improving processes. But FOOM advocates just ignore that and assume that all the problems that matter in real life are actually easy ones where complexity arguments don't apply, somehow.
Sometimes they get sloppy and ignore complexity entirely. If your story about FOOM AI involves it solving NP Hard problems, you should probably rethink your ideas, not the other way around. And yes, I know that P != NP isn't technically proven, but noone seriously doubts it, and if you want to be pedantic, you could substitute something like the Halting Problem, which people often implicitly assume AIs can solve.
There's also this weird obsession with simulations, without any real consideration of the complexity involved. My favorite was the story about a computer that could simulate the entire universe, including iteself with perfect accuracy in faster than real time. But pretty much any time simulations comes up, there's a lot of wooly thinking.
u/None [+3] (a day later)
I don't really know anything about these questions, but my first (and perhaps very naive) reaction to this: isn't the mere possibility that the takeoff could be very fast and the computational problems tractable something to be worried about?
For example, if you were 95% confident that one of your objections here would hold in real life, that still leaves a 5% chance of potential disaster.
u/alexanderwales [+6] Time flies like an arrow (a day later)
In Superintelligence Bostrom argues that medium or fast takeoff is more likely than slow takeoff, a sentiment which is echoed by a fair number of people on LessWrong. There was a recent article by Scott Alexander that said he thinks we live in a world where the jump from infrahuman to superhuman is going to be very fast.
If the argument were "fast takeoff is unlikely but given the risks involved it's still something that we should take seriously" it would be a lot more palatable (though then it might read like Pascal's mugging). Unfortunately, I think there's also a tendency within the LessWrong crowd to first argue that FOOM AI is possible and then treat it as though it's probable, which doesn't do them any favors, especially given the lack of rigor applied to the question of probability.
u/None [+1] (a day later)
There was a recent article by Scott Alexander that said he thinks we live in a world where the jump from infrahuman to superhuman is going to be very fast.
He's entirely wrong about that. Even Eliezer and Bostrom's arguments rely on the AI starting out human-level intelligent, that is, capable of doing the computer-programming tasks necessary to improve itself usefully. A jump from "cow" to "superhuman" is so implausible I'd buy "someone deliberately upgraded it" first.
u/Uncaffeinated [+1] (a day later)
There are a lot of other unlikely but possible disasters to worry about though. What if runaway climate change triggers a feedback loop which makes the earth uninhabitable? What if a new killer disease emerges? What if an asteroid hits the earth?
u/None [+1] (a day later)
We should worry about all of these!
I can't speak for the lesswrong people who are into AI risk research, but I imagine they would say that there are already a lot of people thinking about climate change; NASA is launching a mission to redirect an asteroid; but comparatively fewer people are seriously thinking about AI risk.
u/Vebeltast [+2] You should have expected the bayesian inquisition! (a day later)
but there's no sign of a hyperbolic takeoff, and I don't think there ever will be.
My understanding is that you don't really need hyperbolic takeoff, or even a move move up the computational complexity hierarchy, to get hard a disastrously hard takeoff. All you really need is to move your intelligence off electrochemical computational platforms and onto semiconductors, which gives you something like a factor of 1e8 speedup. Then you accidentally raise a single hyper-fast serial killer without having an equally performant police department in place, and creating that equally performant police department is chicken-and-egg problem.
u/Uncaffeinated [+1] (a day later)
There isn't really any reason to believe in a hard takeoff at all. AI is a large and extremely active field, so there aren't any low hanging fruit. Noone's going to come up with a 1000x improvement overnight.
u/None [+1] (a day later)
Oh, lovely, I've always hoped someone would raise the realistic objections!
The other issue is that it flies in the face of all evidence and theory. Theoretical Computer Science gives us a lot of examples where there are hard limits on self improving processes. But FOOM advocates just ignore that and assume that all the problems that matter in real life are actually easy ones where complexity arguments don't apply, somehow.
I think this is a problem of communication between the theoretical computer scientists (huh, do I count as that?), and the computer-science undergrads, and the general public.
As I recall about NP-completeness for instance, there are many NP-complete problems in which, if an Oracle of some sort gives you 1/3 of the solution, the rest is poly-time computable from that third. Many NP-complete or NP-hard problems can be approximately answered in tractable time. "Best" answers to many questions are intractable, but merely "good" answers are actually pretty easy.
(For example, the non-convex optimization involved in modern deep learning is NP, but as it turns out, most local minima in deep learning loss functions tend to be very near each-other, so we don't actually care which one we get, and stochastic gradient descent unto a local minimum is basically linear-time in the number of samples we learn from.)
The thing is, if you just read through the above, you now know more about computational trade-offs than average, because for some reason we tend not to tell undergrads about those "approximation" thingies.
This is important, since thinking is quite probably conditional simulation and coarse stochastic approximations to true theories can still yield very useful results.
We then get the nasty question of: well, what if your "AI" has a good theory of how to trade-off resources like time and memory for empirical accuracy and precision of its models? Perhaps a theory of decision-making with information-processing costs, cast in terms of the physics that apply to living minds?
In those cases, you certainly can't get some nigh-magical FOOM. But you very likely can get something that is considerably more worrisome because it requires actual expertise to understand and can't be explained neatly to laypeople. Long story short, we often only care about aspects of a problem which can be answered tractably, and we definitely care about tractability when it's a choice between losing a little precision versus gajillions of years of compute-time, and we should assume that halfway-reasonable AIs can carry about the same consideration of trade-offs as us.
if you want to be pedantic, you could substitute something like the Halting Problem, which people often implicitly assume AIs can solve.
The halting problem is actually PAC-learnable, though very difficult.
There's also this weird obsession with simulations, without any real consideration of the complexity involved. My favorite was the story about a computer that could simulate the entire universe, including iteself with perfect accuracy in faster than real time. But pretty much any time simulations comes up, there's a lot of wooly thinking.
Yeah, that's based on Omohundro's "Basic AI Drives" paper, which, at least on the front of, "AIs will want to replace X with a simulation of X", isn't very good. If your AI cares about X in the first place, and X already exists, then it's almost definitely cheaper to obtain information about X by actually observing it than by trying to find principles that allow you to cheaply simulate it with high accuracy (for instance, many sophisticated chemical processes).
So that one's actually wooly thinking and not just lies-to-laypeople.
u/Uncaffeinated [+1] (a day later)
The fact that it is PAC learnable is more of a mathematical curiosity than anything, since all it's really saying is that given a distribution of terminating programs, you can estimate a time bound below which most of them will terminate.
Re approximation: There are some problems where approximation is useful and some where it isn't. Generally, any problem inspired directly by the real world (routing your trucks, optimizing manufacturing processes, etc.) is a problem where approximations are useful. By contrast, more abstract problems, such as anything from cryptography tend to require an exact solution, where approximations are useless.
There also seems to be a conservation of hardness thing. A randomly generated SAT instance is usually easy, but if you take a hard problem, say factorization, and convert it into a SAT, the resulting SAT instance is also intractable. There aren't any free lunches.
To the extent that "increasing intelligence", whatever that means, increases the ability to solve hard problems, then increasing intelligence is at least as hard as every problem which it enables a solution of. Complexity results just don't allow loopholes like that. (You can still do stuff like increase clock speed, since that's just engineering, but you'll quickly run into physical limits there)
u/None [+1] (a day later)
Re approximation: There are some problems where approximation is useful and some where it isn't. Generally, any problem inspired directly by the real world (routing your trucks, optimizing manufacturing processes, etc.) is a problem where approximations are useful. By contrast, more abstract problems, such as anything from cryptography tend to require an exact solution, where approximations are useless.
There also seems to be a conservation of hardness thing. A randomly generated SAT instance is usually easy, but if you take a hard problem, say factorization, and convert it into a SAT, the resulting SAT instance is also intractable. There aren't any free lunches.
Well yes, of course.
To the extent that "increasing intelligence", whatever that means, increases the ability to solve hard problems, then increasing intelligence is at least as hard as every problem which it enables a solution of. Complexity results just don't allow loopholes like that.
I do agree. I just also think that most problems related to the physical world, the ones that decide whether or not intelligence has real-world uses in killing all humans, are mostly problems were increasingly good characterizations (eg: acquiring better scientific theories) and approximations (possibly through specialized methods like building custom ASICs) can be helpful.
If we put this in pseudo-military terms, I don't expect a "war" against a UFAI to be "insta-win" for the AI "because FOOM", but I expect that humanity (lacking its own thoroughly Friendly and operator-controlled AIs) will start about even but suffer a steadily growing disadvantage.
(You can still do stuff like increase clock speed, since that's just engineering, but you'll quickly run into physical limits there)
When you're worried about the relative capability of a dangerous agent to gain advantage over other agents, "just engineering" is all the enemy needs. A real-life UFAI doesn't need any access to Platonic truths or computational superpowers to do very real damage, nor does a real-life operator-controlled AI or FAI need any such things to do its own, more helpful, job competently.
u/Uncaffeinated [+1] (a day later)
But if you don't have hard takeoff, you're unlikely to have just one AI that's relevant. You'll have multiple AIs that are about equal, or maybe the others aren't quite as good.
But if say, Google has a slightly better AI than Apple, that doesn't mean they win everything.
u/None [+1] (a day later)
Yes, that sounds about right to me. But then you get into Darwinian or Marxian pressures from ecological competition/cooperation between AIs, which generally go towards simpler goals -- unless the AIs are properly under human control, in which case they should be able to stably cooperate for their operators' interests.
u/None [+1] (a day later)
Out of curiosity, what is the "general-purpose wager countermeasure?"
u/Vebeltast [+2] You should have expected the bayesian inquisition! (a day later)
The wager depends on hypothetical existence of a thing Y which reinforces you significantly for a belief X, using a huge weight on the reinforcement to balance out the minuscule probability of its existence. The counterargument is to construct an equally-likely hypothetical Y' that reinforces belief not-X in the same way.
This was originally constructed in response to Pascal's wager, the reification being "Yes, but if I believe in God and I'm wrong then Azathoth or Thor will smite me".
u/traverseda [+1] With dread but cautious optimism (a day later)
Precommitment?
u/None [+2] (a day later)
We do tend to act like a bunch of cultish autists. Also, SpaceBattles is all about Truly Ridiculous Lulz: lecturing them about Rationality or Science is like lecturing Ashmodai on Torah. Who are we to tell them to Stop Having Fun?
u/Vebeltast [+1] You should have expected the bayesian inquisition! (a day later)
lecturing them about Rationality or Science is like lecturing Ashmodai on Torah.
See, that's the thing- they respond with hostility even if there's no lecture, even in incredibly not-lulzy-fun threads. Like, you go into the debate subforum and check out the thread about life extending drugs, and you find out that they've spent three entire pages piling on a guy that said he wanted everybody to live forever. It's just weird.
u/None [+2] (a day later)
Shit, really? And here I just thought spacebattles was all about stuff like Shinji and Warhammer 40k.
u/Vebeltast [+2] You should have expected the bayesian inquisition! (a day later)
Nah, there are damn serious threads on there. Like, The Last Angel is one of the single best portrayals I've ever found of AI in literature - it even minor background spoiler - but I know from watching usernames that, if I posted any link back to an LW blog post, the author would be the only person in the thread that didn't join the lynch mob.
(apologies if I missed a sarcasm there... )
u/None [+2] (a day later)
I did not know, at all, that Spacebattles considered anything to be Serious Business.
u/None [+1] (a day later)
And now I'm actually reading The Last Angel and finding it horrifically boring, basically a bland attempt at
HFYSTANDARD SPACEBATTLES FARE - BATTLES IN SPACE - with overdramatized, clearly pulp-scifi-inspired navel-gazing attempts at Deep Meaning that even invoke "souls" and "sins" for spaceship AIs.Oy gevalt. The damned thing thinks it has a soul and kills because it hates. And this is realistic?
u/Vebeltast [+1] You should have expected the bayesian inquisition! (a day later)
Maybe my perception of it is different because I didn't even care that it's focused on humans. All I really saw was that it listed off about five or six civs that hit various different AI extinction risk failure modes - there's a CelestAI, a paperclipper, one that's unexplained so far but that I suspect is going to turn out to be a Whispering Earring, etc - and the one group that pulled it off because they did an upload and kept it as close to normal psychology as possible (even preventing forking) rather than attempting to design a mind from scratch. Which causes problems and seems cheesy, but I wouldn't be surprised is actually relatively realistic. Also doesn't present AI as magical or "has already read the second half of the book" or "beep boop kill all humans".
I'll give you that it's overdramatic and cheesy in places. I don't feel like the AI is one of them, nor that its navel-gazing is that far past what we'd do in that situation.
u/None [+1] (a day later)
Or maybe I just skimmed it really shallowly.
u/Vebeltast [+1] You should have expected the bayesian inquisition! (a day later)
...I was wondering if the "'hate' engraved on each nanoangstrom" thing had really come up that early in the story, yeah.
Also, I don't often recommend this, but I'd suggest possibly going through the "who replied" list and reading the author's comments in the thread. It's possible that some of the things I'm remembering were extracted from author comments and wouldn't appear in the story proper unless he's done an edit pass. Which is a bit of a technical failure in the construction of the story, yes.
u/None [+1] (a day later)
...I was wondering if the "'hate' engraved on each nanoangstrom" thing had really come up that early in the story, yeah.
I read the first forum page, and then skipped to the 143rd page.
u/Rhamni [+13] Aspiring author (7 hours later)
I watched the movie Pan (2015) today. I enjoyed it, but it has one of the worst messages and most irrational villains I have ever seen. Rant
u/SvalbardCaretaker [+18] Mouse Army (7 hours later)
Thats what you get for watching big hollywood productions!
Its fun to dream of a time where all the common media are suffused with the rationality-memeplex. Childrens books about planning fallacies and sunk cost fallacies!
u/ulyssessword [+13] (47 minutes later)
What's a good way to go about giving to charity? The way I see it, here are two parts to the question: how you choose a cause/organisation to support, and how you go about actually supporting it.
For the first, one obvious answer among this group would be some form of effective altruism, and just leave it at that. That leaves me with the question of what to do about groups that I'm personally involved in, or else are relevant only to the local area.
I don't have a good or simple answer for the second half, other than to give money (the currency of caring.) Beyond that, do you guve a lump sum once a year? Wait for some matching opportunity? Automatic monthly ones? Also, volunteering seems like a good idea for some things, mostly akrasia and community building.
u/blazinghand [+7] Chaos Undivided (3 hours later)
I'm not a very charitable person, but there are a couple of causes/organizations I support. I mostly donate to feel good about myself, but also out of some basic duty to donate some nonzero amount of money. I donate about 0.5% of my gross income. Whenever I think about the money I donate, I feel proud of myself. It's also great to talk about. In terms of value for the money, donating to charity is a great way to make yourself feel good.
I donate a small amount of money to Wikipedia every year. I do this because I think Wikipedia is great, and I get a lot of use out of it. Wikipedia needs (I think) about 3 dollars per year per user to operate, so I donate 10 dollars a year and feel pretty good about helping out one of the most useful tools at the level of "I'm doing my part, at least, and covering for a couple less fortunate people".
I also donate a medium amount of money to Doctors Without Borders, who do good. Givewell doesn't find them transparent enough to be a good idea to fully evaluate (compared to AMF or other charities) but gives them a positive review. Doctors Without Borders is often involved in crisis areas, and also helps provide medicine and medical care throughout the world in underdeveloped communities. It's nice to donate to Doctors Without Borders and feel good about myself.
My last donation is a political one, so you can stop here if you want. I donate a moderate amount of money to the American Civil Liberties Union (which I will not link, since it's political). The ACLU is an organization that defends the rights and liberties of Americans in court and by pushing legislature. Traditionally, they advocate for freedom of speech and religion, defending for example anti-war protestors. They also fought on behalf of the Japanese-American internees during WW2, and more recently for the rights of students, homosexuals, and the poor. They're also aggressively against the PATRIOT act, a set of laws that vastly increases the powers of the state and restricts civil rights in order to fight terrorism. I feel like the ACLU is one of the few big organizations fighting to keep America great and free.
u/None [+3] (a day later)
Ask your group if they need money or labor more right now.
u/Gurkenglas [+4] (2 hours later)
Perhaps you can publically commit money to the first matching opportunity that arises for a given rate and recipient. (Which is effectively matching with the reciprocal rate.) Although all these zero-sum moves in a game of charity are kind of silly, and I don't know how to mathematically tell apart "matching" and "taking hostages" and "proposing trade" and "blackmail".
u/thefreegod [+1] (7 days later)
I personally donate 10 percent of my income to patron free fiction on the internet. The stuff I want there to be more of like The Mother of Learning and tales from my D & D campaign. I feel they are less likely to quit half way if they are making money of it.
u/Rhamni [+11] Aspiring author (11 hours later)
SpaceX just landed the first stage of a rocket. Which is pretty cool, since that means you don't have to build a new one every single time you go into space. It's not gonna make space travel cheap, but it's going to bring the price down quite a lot.
u/gbear605 [+8] history’s greatest story (11 hours later)
A Falcon 9 launch currently costs $61 million (according to wikipedia). According to reading on /r/SpaceX, the first stage makes up 75% of the cost. So yes, a rocket launch can now be down to $15 million. ULA, SpaceX's main competitor, costs the US government $380 million per launch.
To be fair, the cost for the government from SpaceX is $130 million instead of $61 million because of regulations and stuff, so a fair comparison would be $15 million versus $175 million. It's a bit of a difference.
u/SvalbardCaretaker [+1] Mouse Army (a day later)
Does anyone know of the science behind speed of adoption of innovations vs costsavings? Eg. is there a law or relationship between magnitude of cost savings vs adoption rate?
I remember from my chemistry years that industrial replacement cycles are on the order of >50 years; 2007 some of the oldest, most inefficient methods for H² or SO⁴ production were on their last factories, which surprised me. I'd have expected the huge costsavings trough increased effiency to have much faster market adoption.
Related to Falcon 9, I'd expect x10 in cost reduction to lead to 90% adoption in 2 build cycles.
u/gbear605 [+3] history’s greatest story (a day later)
Most interesting, is that SpaceX's goal is to be sending up a Falcon 9 every two weeks. At that pace, no other company can near keeping up unless they also go to the reusable model. Because of this, the rate of adoption will be quite high - ignoring the fact that NASA and other regulatory bodies will slow it down. We'll see what happens.
The major question right now is how much refurbishment will need to be done to the first stage after landing it.
u/TaoGaming [+1] No Flair Detected! (2 days later)
I'm not aware of any studies, but the obvious relationship would be based on the required capital investments and rate of return. Effectively ROI. For things like chemical factories I assume regulatory costs of opening a new facility slow the process down by lowering ROI
As older facilities breakdown eventually it becomes worth it to upgrade.
But I suspect if you asked this on marginal revolution you'd get a better answer. Isn't it tabbarok's law -- there are always more studies than you think.
u/Nighzmarquls [+5] (7 hours later)
I just read a very interesting abstract of recent research here.
I consider my understanding of the topic amateur but this seems like potentially a really big deal for understanding brain structure.
u/ulyssessword [+6] (12 hours later)
Is there a word for "something like backlash, but without anything to backlash against"?
The two examples that bring this to my mind are the actress playing Hermione Granger on stage being black, and Canadian Defense Minister Harjit Sajjan being Sihk and East Indian (I'm sure that there are more, and non-race ones as well.). I haven't seen any criticism of them, but I've seen a lot of people loudly proclaiming that they're not only okay with it, but fully supporting it as well.
u/Seeworthy121 [+8] (12 hours later)
forlash?
u/LiteralHeadCannon [+2] (13 hours later)
Sometimes I see the-thing-being-complained-about arise, in such a case... as backlash-to-the-backlash, and then it's used as evidence that the backlash was well-founded to begin with.
u/eaglejarl [+2] (a day later)
The two examples that bring this to my mind are the actress playing Hermione Granger on stage being black,
It took me a full minute to realize that you meant Hermione was being played by a black actress. I thought you meant that Emma Watson was performing in blackface
u/Charlie___ [+1] (4 days later)
Signalling, showing solidarity, rallying, that sort of thing?
u/Rhamni [+1] Aspiring author (21 hours later)
Don't forget the black Stormtrooper in the new Star Wars film! Although there as well, I haven't seen anyone being racist. Just a few people going "Aren't the stormtroopers all clones of that one guy?" and then someone saying "The clone thing didn't work so now they just kidnap babies" and the 'racists' going "Oh, ok."
u/rugurhfurghhrughh [+5] (a day later)
http://strongfemaleprotagonist.com/issue-6/page-5-6/ cross post to /r/getmotivated
u/Rhamni [+5] Aspiring author (a day later)
So hey, if you ever wanted to build a rational dystopia, China has got a few tips for you. Imagine Facebook, if the opinions and financial situation of your facebook friends made it easier or harder for you to get jobs, loans, etc.
u/SvalbardCaretaker [+3] Mouse Army (a day later)
Its been done before! http://www.smbc-comics.com/?id=2286
u/Rhamni [+3] Aspiring author (a day later)
That's neat, although the China thing appears to be actually happening.
u/Gigapode [+1] (2 days later)
I take it all back now. Merry Christmas.
u/rhaps0dy4 [+1] (3 days later)
Check out this paper about quantitative style features of more and less successful literary works:
http://aclweb.org/anthology/D/D13/D13-1181.pdf
I am most surprised by the fact that more successful books are found to use more prepositions.