37 KiB
An omniscient source offers to provide a truthful answer to a single question. What would be the most beneficial question to ask?
-
Author: u/pizzahotdoglover *
-
URL: https://www.reddit.com/r/rational/comments/8t7v1i/an_omniscient_source_offers_to_provide_a_truthful/
-
Score: 14
-
Created: 2018-06-23T05:32:15
Post:
[removed]
Comments:
u/eroticas [+30] (2 minutes later)
"What are the words that I would benefit most from hearing you say"?
u/pizzahotdoglover [+8] (4 minutes later)
Dump your girlfriend, she's cheating on you, and if you stay with her it will ruin your life.
u/Silver_Swift [+16] (an hour later)
Coming from an omniscient source, this would be incredibly reassuring. The answer could have been something like "kill yourself on Juli 31, 2028, because a UFAI comes into existence the day after".
u/pizzahotdoglover [+12] (an hour later)
ROKO'S BASILISK IS REAL. BEGIN WORK ON UFAI IMMEDIATELY.
u/Silver_Swift [+3] (2 hours later)
Ouch, yes that would be way worse.
u/sparr [+7] (8 minutes later)
I
?
u/IntPenDesSwo [+16] (19 minutes later)
#WE
Hammer and sickle fades into background
u/pizzahotdoglover [+9] (25 minutes later)
Yeah, should have asked what words humanity would gain the most benefit from hearing. Fortunately I edited the OP before you guys cheated and broke the universe.
u/eroticas [+7] (an hour later)
Yes, I /u/sparr
I'm maximizing my utility function, not yours, after all ;) such is the goal of any rational agent
I'm altruistic though and a baseline human values so what's best for me is pretty similar to what's best for humanity. I would hazard a guess that what I want is actually more or less identical to what humanity wants but I'm not taking any chances. If humanity does not converge in values then it's still my (coherent, extrapolated) values that I care to maximize. Ordinary one can't do this in real life, one has to work as a coalition, but when the opportunity arises...
It's not my fault /u/pizzahotdoglover decided my utility function was kinda selfish and self involved. The real entity would know that my true utility function was altruistic and more concerned with humanity than my personal life.
u/None [-1] (2 hours later)
[removed]
u/pizzahotdoglover [+3] (2 hours later)
BAD BOT!!! How fucking dare you?!? Fuck off!
u/GoodBot_BadBot [+1] (2 hours later)
Thank you, pizzahotdoglover, for voting on hotdog_bot.
This bot wants to find the best and worst bots on Reddit. You can view results here.
^^Even ^^if ^^I ^^don't ^^reply ^^to ^^your ^^comment, ^^I'm ^^still ^^listening ^^for ^^votes. ^^Check ^^the ^^webpage ^^to ^^see ^^if ^^your ^^vote ^^registered!
u/None [+0] (2 hours later)
[removed]
u/masterax2000 [+2] Chaos Legion (2 hours later)
Bad bot. Worst bot.
u/pizzahotdoglover [+2] (3 hours later)
It's h*tdogs all the way down! This is an ironic sub to almost be the victim of a recursive bot loop.
u/None [+1] (an hour later)
[deleted]
u/None [+1] (an hour later)
[deleted]
u/EliezerYudkowsky [+17] Godric Gryffindor (an hour later)
Will you either answer this question in the negative, or become my good-genie servant for eternity?
u/pizzahotdoglover [+8] (2 hours later)
I will not become your good-genie servant for eternity, so there is no available true answer to this question, since it incorporates a paradox.
u/None [+2] (5 hours later)
I know it involves a paradox but I'm not seeing it so please explain.
u/WarningInsanityBelow [+5] (8 hours later)
The problem is that English contains grammatically correct sentences which when you attempt to use standard logic on will lead to contradiction.
The problem arises when you allow unconstrained referencing of other statements. One solution is to only allow a statement if it either does not reference any other statements, or only references other statements which have already been shown acceptable using one of these two rules.
The problem with Eliezers statement is that it uses the phrase "this question" which references it's self, so it can't be introduced via the first case, and can't be introduced via the second rule because to do so you must have already introduced the statement.
See also the liar paradox or Russell's paradox.
u/pizzahotdoglover [+2] (15 hours later)
You must answer this question truthfully: will you answer in the negative?
If you say yes, it's a lie because you are answering in the affirmative. If you say no, it's a lie because you are falsely claiming you will not answer in the negative when in fact you are. Thus no true answer is available. It's a paradox similar to the statement, "this statement is false" or "I always lie."
u/siIverspawn [+1] (3 months later)
This is what's going on, I think. You have
A = answer this question in the negative
B = become my good-genie servant for eternity
And the question is phrased as either... or, which is the logical connector that is true only if the two statements have different values. (That's the confusing/ambiguous part, since it's different from a simple or connector.) So S = (A ≠ B)
Now, if the omniscient source answers yes, then S is true, so A ≠ B. But A is false, since the answer wasn't negative. Hence B is true, success. If the omniscient source answers no, then S is false, so "A ≠ B" is false, hence A = B. The question was answered in the negative, so A is true, hence B is true, success. Either way, the omniscient source is now your genie.
u/cerebrum [+1] (5 months later)
Is the "either" optional?
u/WarningInsanityBelow [+16] (39 minutes later)
"In ZFC, what is the shortest proof, counter example or proof of undecidability, should they exist, of the following statements insert list of every mathematical problem which we can think to name."
Should be a decent first lower bound.
If vague questions are allowed, something like this would be better:
"From the set of Friendly intelligences which can be reasonably executed on our hardware, what is the source code of the most beneficial one (in a language we actually have)?"
u/pizzahotdoglover [+13] (47 minutes later)
It would answer the first one on the list only. Multi-part questions will be interpreted as separate questions, so anything after the first part will be disregarded.
Your second idea is quite clever, but it wouldn't give you "most beneficial" unless you defined that more specifically.
u/WarningInsanityBelow [+6] (53 minutes later)
Your second idea is quite clever, but it wouldn't give you "most beneficial" unless you defined that more specifically.
I would guess something like average time taken to implement actions in the world which are beneficial to us weighted by gain in utility and inversely weighted by complexity.
u/pizzahotdoglover [+5] (an hour later)
Error- recursive definition of beneficial.
Assume it's like an extremely comprehensive information retrieval system that can access any discrete information but can't make value judgments or do any subjective analysis.
u/WarningInsanityBelow [+4] (an hour later)
Ok, define something as beneficial in my clarification as something which increases utility.
(nitpick: I don't consider my definition to be recursive, since in the first one I used beneficial in a sense of 'degree to which it is good' and the second time in the sense 'whether it is good'. It just so happens by a quirk of English that these concepts have the same word. Though of course your machine wouldn't like either concept since they are both subjective)
u/pizzahotdoglover [+1] (an hour later)
I had another thought about FAI. Are you sure it's such a good idea to create one? Even if you defined its friendliness as carefully as possible, it could still have pretty dramatic and, in retrospect, bad consequences. For example, in The Metamorphosis of Prime Intellect, (MINOR SPOILERS) a FAI bootstraps itself into omnipotence, then does a lot of things that technically achieve the utility function of reducing harm to humans, but in doing so, it uploads everyone to the cloud and doesn't allow anyone to come to harm or die even if they want to, deletes large sections of reality to improve its processing power, and gets kills off all extraterrestrial life, since it might one day threaten humanity, resulting in the total annihilation of tons of sapient species.
Furthermore, even if you did create a FAI so carefully that it would never do any of that stuff, what if it reproduced and its offspring was an asshole? Or what if someone with an incompatible utility value got their hands on its code and made an evil twin? It's a dangerous Pandora's Box to open.
u/vakusdrake [+5] (an hour later)
Even if you defined its friendliness as carefully as possible, it could still have pretty dramatic and, in retrospect, bad consequences.
See all the examples you mention are the results of somewhat obvious failures with regards to its utility function. Also FAI or really nearly any AGI don't create new AI with different utility functions because it might threaten the fulfillment of its own utility function.
u/pizzahotdoglover [+1] (2 hours later)
I'll concede the point that the AI would refrain from reproduction to avoid the possibility of its offspring harming humanity, but I think my other point is valid, even if the examples I cited were obvious. The examples are meant to illustrate that an AI would have very different thought processes than people do, and that no matter how careful we are, it's almost impossible to think of every single possible contingency. I mean, in all the discussions on r/rational about writing utility functions for AI, have you ever seen someone suggest that AI also protect alien life? I'm not saying that if people sat down and actually created one, they wouldn't think of it (after all, the author of the story thought of it), but it's just an example of a blind spot.
I'm suggesting that there are unknown unknowns that no human has ever or would ever conceive of, that could still have devastating consequences if not addressed.
u/ShiranaiWakaranai [+2] (3 hours later)
protect alien life?
There's usually something along the lines of "protect sentient/sapient/intelligent life", which includes any alien life we care about. It can wipe out alien microbes for all we care (unless of course, microbes are sentient).
u/pizzahotdoglover [+1] (3 hours later)
So it destroys 1 billion alien species that would otherwise have evolved into altruistic inventors who would maximize everyone's utility functions a few eons down the road.
u/ShiranaiWakaranai [+3] (3 hours later)
Does it know that those alien species would have maximized everyone's utility functions? If so, it would have let them live, since that is the method of maximizing utility. If not, then there's no real evidence that those alien species would have evolved, so wiping them out is fair game.
After all, every single action could potentially give rise to or prevent some maximally happy outcome, thanks to the butterfly effect. So if you weren't allowed to take actions that could prevent happy outcomes even though you have no evidence to suggest that that is the case, you wouldn't be able to take any action at all.
u/vakusdrake [+2] (4 hours later)
The underlying issue here is that you can make your definition of friendliness include judging friendliness the same way as you, which means it by definition it can't end up inadvertently not friendly by your standards.
u/WarningInsanityBelow [+3] (2 hours later)
A friendly ai wouldn't do anything bad unless this was the least bad available option (modulo knowledge and computational constraints). If it did, it wouldn't be friendly (e.g. intellect prime is not friendly). Unfortunately we don't have any precise definitions of friendly, this is the reason why I thought you wouldn't allow my second question.
u/pizzahotdoglover [+1] (2 hours later)
I would say that with its omniscience it would be aware of everything ever said or written that defines "Friendly AI" and, combined with its knowledge of you, come up with a definition that fits your best understanding of what it means. And as I said in my response to /u/vakusdrake's comment, there may be things that would never occur to any human to include in the definition of FAI, that would nevertheless have serious negative consequences (one of the examples I gave was of a FAI annihilating all alien life because one of its directives was to protect human life).
u/None [+12] (40 minutes later)
We already know that the answer is 42, why wait another few million years?
u/phylogenik [+10] (33 minutes later)
pfft re: your edit far worse can be done
Please utter the sequence of values whose utterance in response to this question will globally maximize my utility function... within my future light cone... averaged across all Everett branches?
I'd reckon the next most likely question would be something like "what is the shortest (but extremely well documented and commented?) source code written in an existing programming language and capable of being compiled into a program executable on existing hardware that will, in the shortest amount of time, bring into existence a recursively improving general artificial intelligence whose existence will maximally satisfy my values and whose values are maximally aligned with my own" or something lol idk
edit: haha called it! ;p
u/pizzahotdoglover [+6] (53 minutes later)
I was going to say you'd have to specifically define your values and what your utility function is, but of course, it's omniscient, so it would already know that information.
I'm also considering a bit limit, since the point of the question is basically, what information has the most value per bit.
u/phylogenik [+3] (an hour later)
I think the bit limit wouldn't do anything to prevent the first sort of question -- though, now that I think about it, with a sufficiently small limit there's no "guarantee" that the "do what I mean" sorts of questions are the best to ask, right? Since they're noncontextual, and I can't imagine my behavior changing substantially with the receipt of any of, say, 2^10 possible ordered sets of 10 bits? Unless maybe I make it -- e.g., say I remain blind to the content of the answer, and then when I have some big, uncertain decision to make, I specify and designate my binary options (in the excluded middle sense, doing something and not doing something) and then "uncover" one of the bits and blindly do whatever action (or inaction) it corresponds to.
Trivially, I could make some $ on high stake roulette, or less trivially become head of state or something and use it to decide whether to wage war. I don't think I could "reuse" bits, even if they fade from conscious memory, since doing so would couple decisions and have to average utility across that (potentially suboptimal) coupling. I could maybe even force certain outcomes if I precommit to doing something really preference-frustrating in the event of the outcome I don't want? or maybe not, actually.
u/pizzahotdoglover [+2] (an hour later)
I edited the OP to exclude AI source codes, since that is basically the "wish for more wishes" answer to the prompt. But that wouldn't exclude questions that would assist with the creation of FAI, like "what currently unknown computer programming concept or development, if explained today, would most reduce the time it takes us to create a FAI?"
u/Tommy2255 [+4] (56 minutes later)
Clarification request: Under what circumstances do I have this opportunity to question this omniscient oracle, and what form does the answer take?
If I have to ask the question on the spot without preparation, and the answer is an immediate verbal response, then that would have a major impact on the people asking the oracle to write software for them. I doubt any ordinary human can memorize the complete code for an artificial intelligence after hearing it read aloud once.
u/pizzahotdoglover [+3] (an hour later)
There is no time limit on when you have to ask the question, and it will be given to you in any format you choose, including digital/searchable. So if you wanted you could hold a worldwide summit of scientists and world leaders to spend years debating or refining the question, or you could just ask it right now if you really did have a shot with Mary from your 11th grade class.
u/GCU_JustTesting [+4] (2 hours later)
Does P=NP
u/sicutumbo [+3] (an hour later)
"Give me the proof or refutation of P=NP."
I think this is at least a good answer to the question given your restraints, if not absolutely optimal. It has a concrete answer and isn't asking to solve all my problems for me, but can still in effect help do so through giving the algorithm for solving any mathematical proof and giving the avenue to make basically all programming (excluding the AI ethics bits) problems trivial.
Well, it would suck if P doesn't equal NP, or the general solution requires a googol operations, but the potential reward is so high that you can risk merely learning an interesting piece of mathematical knowledge and getting the million dollar Millennium Prize money.
u/ShiranaiWakaranai [+2] (2 hours later)
Though if you're in it for the money, you might as well just ask for lottery numbers.
u/pizzahotdoglover [+2] (3 hours later)
Or the location of valuable undiscovered natural resources, or the chemical formula of a substance that can cure ___.
u/pizzahotdoglover [+1] (an hour later)
This was the first answer I was expecting, actually. And if it tells you that P doesn't equal NP then at least we know that and can avoid wasting resources on the question. There's probably a lot of other implications to knowing that for sure that would be helpful in ways I haven't thought about.
u/vakusdrake [+3] (2 hours later)
Edit: Multi-part questions will be interpreted as separate questions, so anything after the first part will be disregarded.
This doesn't really work as a limitation. Just specify a question whose answer must necessarily include the answers to any other questions you want answered, thus meaning your only real limit here is needing to generate all your questions up-front.
I'd also like to point out that the previously mentioned question "What are the words that I would benefit most from hearing you say?" would very nearly work. However you would need to add the caveat that "benefit" is defined based on your current utility function.
The question works because without a limit on answer length the best answer for it to give you would effectively function as Path to Victory, in fact since it's able to exploit the butterfly effect it would probably actually be vastly superior to PtV. So the most likely outcome could be it causing you to take a bunch of bizzare random seeming actions that lead to the development of a FAI with your utility function happening in a few years due to many different freak accidents.
u/pizzahotdoglover [+1] (2 hours later)
Assume that it's smart enough to work around semantic traps. If the question by its phrasing necessarily includes the answers to A, B, and C, it will identify this and only answer A. If that's not possible, it would return an error- too much information requested- ask a single question.
u/vakusdrake [+2] (4 hours later)
Again that doesn't work unless it just barrs any question which outputs too much information. There's no coherent way to distinguish whether a question is made of smaller component question, because nearly any question can be presented as multiple smaller questions.
u/pizzahotdoglover [+1] (15 hours later)
The entity makes a judgment call with its omniscience
u/vakusdrake [+2] (a day later)
The issue is that the distinction between complex questions and multiple questions is kind of nonexistent. So two people could easily come up with the same question, with only one of them having constructed it out of multiple smaller questions and other having developed it from scratch.
u/pizzahotdoglover [+1] (a day later)
Right. So if you choose to ask an extremely narrow question, which could have been answered as part of a more complex, acceptable question, then you will have squandered your opportunity. On the other hand, if you construct a question that is basically a multi-part question, you won't get a multi-part answer. The entity will decide whether this is the case.
So for example, if you asked, how does photosynthesis work, the entity could tell you that process, even though it contains more than one piece of information. But if you ask, (a) how do leaves absorb sunlight energy, and (b) how do plants spend absorbed energy, then you will have phrased your question foolishly, because you'll only get an answer to either (a) or (b).
On the other hand, if you ask it to fully recount all information on plants, that will be judged too broad of a question, even though it only requested one "thing" semantically.
u/vakusdrake [+2] (a day later)
Right. So if you choose to ask an extremely narrow question, which could have been answered as part of a more complex, acceptable question, then you've squandered the opportunity. On the other hand, if you construct a question that is basically a multi-part question, you won't get a multi-part answer. The entity will decide whether this is the case.
I'm saying there's fundamentally no metric it could use to determine whether something seems like a multi-part question and the metric you seem to be using is just whether it sounds like a multi-part question to you. However that metric can be trivially subverted by just phrasing your questions better.
u/pizzahotdoglover [+1] (a day later)
I get that, and I'm saying it has enough omniscience to make a judgment call, just like you or I could. If you agree that it can interpret things like utility functions and whether an AI is friendly or beneficial, then you should also agree that it knows enough about multipart questions and semantics to make a judgment call on whether a question qualifies. At some point, such a judgment call is necessary, otherwise that defeats the entire limitation of the "single question". If multi part questions were allowed, then you could just ask it unlimited questions by using clever phrasing.
u/vakusdrake [+2] (a day later)
Right I'm just saying that you aren't using a consistent standard either so saying it uses the same standard as you (with your FAI analogy) doesn't fix anything.
Comparing "determining whether something is actually multiple questions" to friendliness doesn't really work, because it implies that there is actually some non-arbitrary metric (as in not just whatever is currently your whim) being used even if you can't articulate it/understand it without omniscience.
u/pizzahotdoglover [+1] (a day later)
How would you suggest that the "one question" restriction be enforced, if you were in charge of imposing that restriction? I agree that my method is imperfect, but it's the best way I can think of.
u/vakusdrake [+2] (a day later)
Honestly I would probably just impose a limit in the number of bits that could be transmitted. However that creates the obvious problem that while that limit may be trivial for an omniscient being to know, for us knowing the exact number of bits contained within a given question is practically a intractable problem.
u/pizzahotdoglover [+1] (a day later)
I wonder how you would optimize a yes or no question with a guaranteed truthful answer?
u/vakusdrake [+2] (a day later)
Hmm that gives me an idea.. I don't actually think you could do anything very useful with just a single yes-no question (at least if you had no way of proving to others this happened).
However I think you could probably make the oracle useful if you simply limited it to giving you some finite number (say 20) of yes no question. Yes it would sort of change the premise however it would also eliminate the problems that the limitation on multiple merged questions was designed to deal with in the first case.
Additional limits may include having to merge the X# questions into a single question, or forcing people to ask their questions all within some short timespan. This would allow people to take their time coming up with good questions but not let them employ many additional exploits available to them if they could space out their questions over any amount of time.
P.S. If you were going with this and you wanted to actually have it be on the same level of usefulness as you probably had in mind with regards to the original scenario you'd probably want to give people a fair deal more than twenty questions.
u/ArmokGoB [+3] (2 hours later)
"What message when posted online and linked where I will link it, will make the vast majority of humanity completely dedicate themselves to a non-counterproductive policy that maximizes the probability of FAI within the next 100 years."
u/ShiranaiWakaranai [+2] (3 hours later)
This seems dangerous, because maybe the policy that maximizes the probability of an FAI is to spur people into recklessly rushing out AIs, and so also increase the probability of a UFAI.
E.g. Maybe before your message, the probability of FAI in 100 years is 10%, UFAI is 20%, and no AI is 70%. There could be a message advocating careful coding that leads to 15% FAI, 5% UFAI and 80% no AI, which is what you would want. But then there could be a message advocating rushed coding that leads to 20% FAI 80% UFAI, which has a higher FAI probability and so is the message you are given.
u/ArmokGoB [+2] (5 hours later)
That's what "non-counterproductive" means. Also, we seem to vastly disagree what the probabilities before the message is; I'd say closer to 2% FAI, 1% no AI, and 97% UFAI. That last one split into something like 88% everyone simply dies and the future value of the universe is exactly 0, and 9% something unimaginablly malevolent with a million times more suffering than the worst hells ever imagined by humanity.
u/ShiranaiWakaranai [+2] (12 hours later)
That's what "non-counterproductive" means.
What exactly does that mean though, quantitatively? Is any policy that increases the chance of a UFAI considered counterproductive? Is there some ratio threshold of FAI to UFAI probability that a policy must have to be non-counterproductive? If the restrictions are too tight, you might end up with policies that have very weak effects that barely change any of the probabilities.
Also, we seem to vastly disagree what the probabilities before the message is;
Eh, they were just numbers I chose to illustrate the problem. My real opinion is 0% FAI 99% UFAI 1% some catastrophic event(s) wipes out all/most of humanity before they build a UFAI, simply because I don't believe FAIs are possible. I can't use this for the example since every policy would have no effect on the probability of an FAI.
u/ArmokGoB [+2] (13 hours later)
Yea, that could happen, but it seem unlikely given my priors. It's not perfect by definition, anything that is would be against the spirit of the rules.
u/pizzahotdoglover [+1] (2 hours later)
You should change "vast majority" to "highest possible fraction" or something, to avoid the answer, "no such message exists."
u/ArmokGoB [+2] (5 hours later)
Maybe, but if I do that it contains multiple lose variables and underspecifies how to do tradeoff between them
u/brbrainerd [+2] (2 hours later)
I am assuming an anthropocentric metric of benefit for the sake of time, brevity, and to avoid obvious monkey's paws.
My top choices are:
- "Provide a complete Standard Model such that it accounts for as many phenomena as possible with the highest possible degree of accuracy." In addition to gravity, dark matter, and energy this takes care of any unobservables that we would otherwise never be able to fully understand.
- "What is the genetic code of an organism that would provide the greatest benefit to humanity?"
- "What is a safe method to optimize human intelligence?"
- "What series of actions can we reasonably perform that will maximize the long-term probability of humanity's satisfaction and survival?" If it is possible to survive in perpetuity (e.g. avoiding heat death) these answers will be preferentially selected. If our extinction is inevitable we don't waste an answer on a response like "you can't."
If necessary, we can avoid being given answers we could never use by adding to the quotes above: "...that humanity will have a 100% chance of utilizing to our greatest benefit before extinction, the end of the universe, or a maximum of Graham's number of years, whichever is soonest." Probability takes care of failure during construction from all sources, so finding the most beneficial option requires playing the long game, but not so long that the universe dies before we have a few billion years to benefit from the results. If heat death ends up not being the inevitable fate of everything, perhaps as a result of the answer we receive, setting an arbitrarily high duration eliminates responses that would have the highest theoretical benefit but could not be fully realized in a finite amount of time.
Edit: u/erotica's answer takes the prize in my opinion, but hopefully this will provide you with a few more specific examples to think about :).
u/pizzahotdoglover [+3] (3 hours later)
You should add a caveat such that if no such model exists, provide the model that accounts for the most possible phenomena.
I was thinking about this one when other people were asking for AI source code. After all, the intelligence doesn't have to be a computer. But it would be tragic if we never developed the technology to actualize the genetic code into a healthy organism. And it'd be hilarious if when we did, it just turned out to be Jesus.
Stay in school, kids!
It could tell you the single most beneficial action or the first action in the series you requested, but asking for the whole series would count as a multi-part question.
u/brbrainerd [+2] (3 hours later)
You should add a caveat such that if no such model exists, provide the model that accounts for the most possible phenomena.
Good catch. I also added "to the highest possible degree of accuracy," though it is possible that we would receive a highly probabilistic model with less than optimal utility (not unlike the model we have today ;) ).
But it would be tragic if we never developed the technology to actualize the genetic code into a healthy organism.
I think the final paragraph takes care of that.
And it'd be hilarious if when we did, it just turned out to be Jesus.
Despite my (lack of) religious beliefs, I would watch the hell out of that sci-fi.
asking for the whole series would count as a multi-part question.
Perhaps asking for the most beneficial overall strategy, instead of a rote series of actions, would result in a succinct but complete answer?
u/pizzahotdoglover [+2] (3 hours later)
Despite my (lack of) religious beliefs, I would watch the hell out of that sci-fi.
Lol yeah, that's actually how the Second Coming of Jesus comes about. Who knew?
Perhaps asking for the most beneficial overall strategy, instead of a rote series of actions, would result in a succinct but complete answer?
That would definitely work.
u/ShiranaiWakaranai [+3] (3 hours later)
"What series of actions can we reasonably perform that will maximize the long-term probability of humanity's satisfaction and survival?" If it is possible to survive in perpetuity (e.g. avoiding heat death) these answers will be preferentially selected. If our extinction is inevitable we don't waste an answer on a response like "you can't."
Suppose the omniscient being does give you a correct answer for this. How would you convince the rest of humanity to follow those actions though? You can't really prove that you got the answer from an omniscient being, since it disappeared after you asked it that one question.
u/pizzahotdoglover [+2] (3 hours later)
Sounds like /u/brbrainerd would be the tragic love child of Cassandra and Accord.
u/brbrainerd [+2] (3 hours later)
If that's unaccounted for by my use of the term "reasonable," then I believe the probability failsafe in the final paragraph will steer us around this issue. Instructions that are unpersuasive or otherwise non-communicable would necessarily have a low probability of overall success.
u/pizzahotdoglover [+1] (3 hours later)
I think for the sake of the prompt, we can assume that people will be aware of the omniscient entity's offer and omniscience. Otherwise, most answers would be, Step 1: Become dictator...
u/ShiranaiWakaranai [+2] (2 hours later)
"What is the code for a program that will answer any question I ask of it correctly (if it complies with the above rules)?"
Such a program exists, because you can simply program a massive look-up table for every possible question with their answers as coded constants. It is not an AI, because it isn't smart, it's just looking up a table. It isn't a multi-part question. It is an extremely narrow question, because the code given either works or does not. So it should comply with all rules and thus result in getting all answers to all questions that comply with the rules.
u/pizzahotdoglover [+1] (3 hours later)
Congratulations, you've munchkined your way into unlimited knowledge. Here is your infinitely long code. It may take some time to enter into your computers and compile. But really, since it includes the answers to every possible question, I think this would just be interpreted as a multi-part question. In other words the restriction is less, "no multi-part questions" as it is, "no questions that require numerous answers."
u/None [+2] (5 hours later)
As others have mentioned, that is not a coherent restriction. Or at least making it coherent is non trivial, how do you distinguish a single answer? Many questions can be broken up into simpler ones, and a way to demarcate questions that would include giving the source code for a FAI, but exclude the other answers to me seems like it'll be highly contrived.
u/pizzahotdoglover [+3] (15 hours later)
As I mentioned, the entity makes a judgment call.
u/ShiranaiWakaranai [+2] (2 hours later)
"How can I acquire as much knowledge as possible after I ask this question?"
Should hopefully result in something along the lines of:
"By listening very carefully to the following information:"
u/None [+2] (3 hours later)
[deleted]
u/ShiranaiWakaranai [+2] (3 hours later)
I guarantee your answer will be along the lines of "You can't."
u/None [+2] (3 hours later)
[deleted]
u/ShiranaiWakaranai [+2] (3 hours later)
But according to the rules, we're supposed to ignore the existence of the omniscient being for our question.
u/None [+2] (5 hours later)
How do I build HLMI?
How do you align arbitrary level artificial intelligence with human goals? (Alignment problem)I'm sure there are several other "impossible" problems which if you knew the answer to would change life as we know it.
u/King_of_Men [+3] (38 minutes later)
"How can I become omniscient myself?"
u/WarningInsanityBelow [+12] (41 minutes later)
The answer might just be: "you can't"
u/pizzahotdoglover [+7] (51 minutes later)
Or, "what could I say to persuade you to answer additional questions?" Still risky, because the answer might be, "there is no way to do that."
u/sicutumbo [+5] (an hour later)
Or it might tell you, then disappear before you can do anything with the information
u/pizzahotdoglover [+2] (an hour later)
Wait come back! Fuck. I should've asked for the Grand Unified Theory of Everything.