30 KiB
[BST] Orange and Blue Morality: help me make an "anti-rational" faction work in Rationalist fiction
-
Author: u/notmy2ndopinion Concent of Saunt Edhar*
-
URL: https://www.reddit.com/r/rational/comments/3r5kur/bst_orange_and_blue_morality_help_me_make_an/
-
Score: 15
-
Created: 2015-11-02T01:29:57
Post:
I'm working on the rationalist characters for my proto-biopunk D&Desque themed setting, entitled "Medics and Magics." It basically takes place in the 1800s with fantasy races as humans with extra sensory abilities, rebuilding society after the crushing collapse of the magical world. The main team of characters are rationalist scientists, chemists, and medics who are sometimes seen as 'witches, wizards, and warlocks' so to speak because the general public fears their powers.
One of the main factions that is organizing against them is the Council of Mystics who are actively hunting down these scientists because... uh...
Here's where I'm running into a bit of a road block. Scientists have a bad reputation because of horrific experiments in the past (think of Twig but more medieval... or the Duke of Gibea in the Kingkiller chronicles) as well as the creation of several existential threats like an unfriendly magical AI, grey goo, nuclear weapons, and bioterrorism.
I'm thinking that the Council of Mystics are a rebel group that initially started off like the SCP, the Men in Black, or the Ita (from Anathem) that basically fought off weird threats, destroyed tomes of power, fought against magic... and won. When magic disappeared, civilization plunged into the Dark Ages.
Here's the rub. They were an organization that made sense back in the day when they had to the the countervailing force against wizards. They destroyed knowledge that could destroy the world (when it was in the hands of the wrong people.) But now, the world is moving into a renaissance spurred primarily by the new races. The Mystics are persecuting my rational protagonists through a witch hunt, which initially seemed like an anti-rational or fear-mongering stance.
I don't want the Mystics to come off as stupid or evil. (See sidebar:)
Nothing happens solely because 'the plot requires it'. If characters do (or don't do) something, there must be a plausible reason. Any factions are defined and driven into conflict by their beliefs and values, not just by being "good" or "evil".
Any factions are defined and driven into conflict by their beliefs and values, not just by being "good" or "evil".
Should I just keep writing and let the background for the setting unfold?
Any ideas on how to make them follow another axis of alignment? Something stronger than irrational vs rational, or mystery vs explanation. Conservative vs Progressive hits the wrong tones for my setting too.
Thanks
Comments:
u/alexanderwales [+18] Time flies like an arrow (50 minutes later)
Magic and its consequences have been a disaster for the human race. They have greatly increased the life-expectancy of those of us who live in “advanced” countries, but they have destabilized society, have made life unfulfilling, have subjected human beings to indignities, have led to widespread psychological suffering (in the Third World to physical suffering as well) and have inflicted severe damage on the natural world. The continued development and use of magic will worsen the situation. It will certainly subject human beings to greater indignities and inflict greater damage on the natural world, it will probably lead to greater social disruption and psychological suffering, and it may lead to increased physical suffering even in “advanced” countries.
The industrial-technological-magic system may survive or it may break down. If it survives, it may eventually achieve a low level of physical and psychological suffering, but only after passing through a long and very painful period of adjustment and only at the cost of permanently reducing human beings and many other living organisms to engineered products and mere cogs in the social machine. Furthermore, if the system survives, the consequences will be inevitable: There is no way of reforming or modifying the system so as to prevent it from depriving people of dignity and autonomy.
If the system breaks down the consequences will still be very painful. But the bigger the system grows the more disastrous the results of its breakdown will be, so if it is to break down it had best break down sooner rather than later.
We therefore advocate a revolution against magic. This revolution may or may not make use of violence; it may be sudden or it may be a relatively gradual process spanning a few decades. We can’t predict any of that. But we do outline in a very general way the measures that those who hate magic should take in order to prepare the way for a revolution against that form of society. This is not to be a political revolution. Its object will be to overthrow not governments but the magical basis of the present society.
In this article we give attention to only some of the negative developments that have grown out of the system. Other such developments we mention only briefly or ignore altogether. This does not mean that we regard these other developments as unimportant. For practical reasons we have to confine our discussion to areas that have received insufficient public attention or in which we have something new to say. For example, since there are well-developed environmental and wilderness movements, we have written very little about environmental degradation or the destruction of wild nature, even though we consider these to be highly important.
Directly adapted from The Industrial Society and its Future, AKA The Unabomber Manifesto. Of course, the Unabomber was arguing against a large and pervasive system which society widely agreed upon; not sure how much massaging this worldview would need in order to make it work with what you're writing.
u/xamueljones [+11] My arch-enemy is entropy (3 hours later)
So you just replaced all mention of the Industrial Revolution with magic? That's brilliant!
I was at first wondering how you managed to write up a dense piece so quickly, but still have it sound like a speech written by a professional.
u/None [+10] (12 hours later)
I feel like this really ought to be boldly tagged with, "UNABOMBER MANIFESTO, DO NOT STEELMAN."
u/alexanderwales [+7] Time flies like an arrow (14 hours later)
Here's Bill Joy (co-founder of Sun Microsystems) on the Unabomber, from his essay "Why the Future Doesn't Need Us":
Kaczynski's actions were murderous and, in my view, criminally insane. He is clearly a Luddite, but simply saying this does not dismiss his argument; as difficult as it is for me to acknowledge, I saw some merit in the reasoning in this single passage. I felt compelled to confront it.
I generally agree with that. Just because someone kills a bunch of people doesn't mean that you can immediately dismiss their arguments out of hand.
u/None [+2] (16 hours later)
Well, I of course believe the Unabomber was not only wrong to bomb people but wrong about society in general. Problem is, explaining how takes a lot longer than "steelmanning" his propaganda into something "reasonable" enough to actually persuade people.
As a community, we should probably have a term for things subject to the "simple, easy, and wrong" asymmetry. That's part of why arguments and persuasion are so untrustworthy as guides to truth in the first place.
u/alexanderwales [+9] Time flies like an arrow (16 hours later)
I think that if you want to find a compelling intellectual or moral conflict, you need to find places where there's a reasonable argument that takes a wrong turn somewhere. If it's just someone being evil for the sake of evil and the hero is doing something that everyone agrees is right and just ... that's boring. But if the hero has to untangle what the villain is saying and separate out things which are true from things which only seem true, then you have a conflict worth reading about.
Maybe the principle should be "don't steelman the Unabomber unless you're also going to tear that steel apart"? Because I think if you just shy away from those "simple, easy, and wrong" arguments then you're giving up a large amount of what makes rational fiction worth reading. (I would place both Equestria Online and Quirrell in the camp of "simple, easy, and wrong". But that's part of what makes FiO and HPMOR worth reading.)
u/None [+3] (17 hours later)
Oh, I'd meant "Don't steelman the Unabomber in real life". The corrolary for fiction would be "Don't steelman the Unabomber so much that he looks like the hero."
u/philip1201 [+1] (a day later)
I'm confused. Are you talking about steelmanning in public? I've typically only heard of using steelmanning as a personal practice, and it seems unlikely that a human could write a memetic trap which will convince people when you steelman it and isn't therefore simply correct.
u/None [+2] (a day later)
it seems unlikely that a human could write a memetic trap which will convince people when you steelman it and isn't therefore simply correct.
Then how did the Unabomber, Osama bin Laden, Hitler, and Stalin all happen? Oh, and Abu Bakr al-Baghdadi? And the fictional "rational" villains listed above, whom numerous people across the interwebs have actually been persuaded by?
Insofar as "steelmanning" means "finding the most plausible/probable phrasing of an argument or opinion", it merely consists in the particular kind of lying-with-literal-truths in which you highlight coherent-but-unlikely possibilities, and then let confirmation bias and the availability heuristic persuade your victim.
Emphasis on victim.
u/philip1201 [+1] (a day later)
That sentence referred to 'steelmanning' as I understand it: personal steelmanning. Not sharing with others or trying to convince them, but thinking up the best arguments you can against yourself. Deliberate psychological manipulation isn't supposed to be part of it because it isn't a good argument.
We don't seem to disagree in the territory, you're just using a broader definition for 'steelmanning' than I.
u/None [+1] (a day later)
Not sharing with others or trying to convince them, but thinking up the best arguments you can against yourself.
This is still subject to the same cognitive biases as when "arguing" to someone else, is my view. You can argue yourself into or out of any view you want to hold or not hold.
u/philip1201 [+1] (a day later)
Can you? I can't think of any non-fictional case.
u/None [+1] (a day later)
There are people who convince themselves to go join ISIS on the basis of YouTube videos. I mean, fuck, there are professional apologists for Creationism.
u/philip1201 [+1] (a day later)
Those are convinced by others and try to convince others respectively. They aren't arguing themselves into or out of views.
u/None [+1] (a day later)
No, a lot of them actually direct their own indoctrination without an external persuader.
u/philip1201 [+1] (a day later)
Do you have evidence for that?
Perhaps more importantly, though, it seems to me that if [applying rationality technique] to something can lead you astray, you're doing the technique wrong. Therefore "Don't steelman the Unabomber" => "Don't steelman", for any two common and identical definitions of 'steelmanning'.
It seems like the lesswrong and CFAR definitions ought to be safely applicable to anything. If you expect to be convinced the Unabomber was right after thinking about it a lot, you believe the Unabomber was right.
u/None [+1] (a day later)
When techniques and results conflict, you modify the technique.
u/philip1201 [+1] (2 days later)
Aye, and "don't steelman this one particular guy because you might actually be convinced" is indicitative of failure of the technique.
u/None [+1] (2 days later)
Well yes, which is why I would say, "Don't always steelman: sometimes the apparent or even hidden weaknesses of someone's idea really are weaknesses, and you want to keep weaknesses and uncertainties where you can see them, lest they sneak up and bite you."
u/philip1201 [+1] (2 days later)
Why would you ever want to use a broken technique? Why would you ever want to hide information in a cognitive technique? When, using your definition of a steelman, would it be appropriate to steelman?
Steelmanning (as I'm familiar with it) doesn't remove or 'hide' weaknesses and uncertainties, not that I would know how to hide things from myself through mere thought. Every time you decide not to look for evidence against the steelmanned position, you can infer the existence of unknown missing evidence, which undermines the steelman. (Hypothetical detractors would be able to say "your method of gathering evidence was biased!", which is an unnecessary flaw. Actual detractors may not know that your methods were biased, but that would mean you're consciously leveraging your undeserved credibility, in which case "steelmanning" would include bribing recognised authorities to lie for you, and your credibility would hopefully run out quickly).
u/notmy2ndopinion [+1] Concent of Saunt Edhar (a day later)
Or if you're GRRM, don't give your despicable characters such realistic backgrounds or have them suffer so horribly that you actually start to like them in spite of who they used to be.
u/Transfuturist [+2] Carthago delenda est. (a day later)
If you're GRRM, that's exactly what you should do.
u/notmy2ndopinion [+6] Concent of Saunt Edhar (a day later)
Reading this manifesto, realizing where it is coming from, agreeing with its sentiment, yet being horrified by the actions of its writer... that's precisely the tone I wanted to strike with some of the propaganda in my universe!
Thank you for helping me viscerally feel the guilty curiosity that some of my characters will have when they come across it.
u/sidhe3141 [+11] (2 hours later)
It sounds like you've already got a decent reason for them to oppose the scientists.
These people created the Mad God, they built the World-Ender Engine, they armed the kings with devices that created places that are still wasteland and plagues that destroyed entire countries. And you want to trust them with the power to reshape your very body? Are you nuts?
As for alignment, cutting off some of the baggage attached to Conservative vs Progressive sounds like it could work. Safety vs Change, maybe.
u/notmy2ndopinion [+2] Concent of Saunt Edhar (a day later)
I don't recognize your quote, but I like it!
Safety vs Change sounds close. Security vs Risk? Bah. Perhaps I'm weighing myself down with an imaginary alignment axis that likely won't even make its way into the text.
u/sidhe3141 [+2] (a day later)
Quote is something I made up that seems like it would belong in the mouth of someone in that faction. Go ahead and use it if you want.
u/Transfuturist [+2] Carthago delenda est. (a day later)
Safety vs. Change is Conservatism vs. Progressivism.
u/Geminii27 [+18] (an hour later)
Flip it on its head. What are some downsides of rationalism (and in particular extreme rationalism) that might make people scared or at least wary of rationalists?
- apparent lack of emotion at times;
- strange and unfamiliar thought processes, sometimes leading to strange actions;
- strange language (jargon);
- unwillingness to follow nonrationalists blindly and/or without question;
- arguing amongst themselves;
- somehow, despite all this, they somehow keep coming out on top of situations a lot more than than they should be able to;
- somehow 'magically' prepared to handle things they could not possibly have known were going to happen; and
- normal people who have been taught or trained in their ways, or even just spent a degree of time with them, start exhibiting the same behavioral oddities (including arguing with authority). This is particularly the case with younger adults and children.
So it's quite possible that from the outside, rationalists look like magically empowered humans or perhaps even something inhuman disguised as humans. They're too lucky, too successful, and they speak a strange language. If they collect and share knowledge, they can create or repair powerful and complex tools, know how to make alchemical miracles, are powerful healers, and are sometimes supernaturally successful military leaders/advisers, rulers, and spymasters. And they can turn normal people into more of them just by talking.
And yet, whenever the Mystics confront them, they deny having any kind of supernatural abilities or artifacts, or contact with dark forces. They defy rightful commands, refuse to submit to the Mystic's authority, and insist in the face of all evidence that they wield no Powers, even under torture. It is the sworn duty of the Council to protect the people of the land from demonic infections, and no less so when those influences are subtle and walk under the sun as do men, concealing their poisons as naught but light words and parlor tricks.
This creeping blight of the soul, this rot of mind and wit, must be purged wherever it is found. It cannot be allowed to take root in this land while good hearts stand true.
u/None [+4] (12 hours later)
This creeping blight of the soul, this rot of mind and wit, must be purged wherever it is found. It cannot be allowed to take root in this land while good hearts stand true.
I'm guessing The Emperor Protects?
u/notmy2ndopinion [+3] Concent of Saunt Edhar (a day later)
While I'd love to play up these traits/stereotypes, I don't want people to be 'turned off' the way that some people saw HPJEV as a pratty, precocious know-it-all.
I do really like the idea that stubborn people will refuse to listen because they will be able to turn you into them just by listening.
u/Geminii27 [+3] (a day later)
The rationalists don't necessarily have to conform to the perceptions of them that the opposing faction has. It might be enough that the opposing faction perceives them that way, or that they have been taught to do so by people who have their own agendas or who feel threatened by the rationalists.
Yes, it might all collapse if the Mystics were themselves rational and carefully investigated the rationalists first-hand, but the premise is kind of them not being rational, just being... well, human. Even, quite possibly, well-meaning (as they see it). This makes the conflict less Rationalist vs Mystic and more Thought vs Belief - essentially, one of the core rationalist issues writ large in the metaphor of the groups' inevitable clashes.
The rationalists don't even have to win all the encounters, exactly - convincing people to be rational can be a long and drawn-out process, with many retreats to behaviors driven by fear or tradition. And then there's the fact that even when one side considers themselves to have 'won' a particular battle, the other side may not necessarily have considered themselves to have lost - they're not precisely opposed in their goals, and are playing on somewhat different (if often overlapping) playing fields. I can't see the rationalists as being rabidly anti-Mystic, for instance, and may even admire the Mystics' grim dedication to protecting people, even if the way they go about it tends to interfere with the upgrading of civilization.
It should be quite easy, for instance, to write a story from the perspective of an heroic Mystic who had been raised to be wary of the Rationalist menace (amongst many other things), but had never actually seen one in the flesh, as it were, and wound up separated from the Council for a time, meeting and talking to people who seemed friendly and curious about everything.
u/embrodski [+6] (16 hours later)
Possibly relevant: Epistemic Learned Helplessness.
An EXTREMELY shortened summary that does not do the original post justice, you really should go read it:
I will trust the evidence ... I think the average high school dropout both doesn't and shouldn't. Anyone anywhere - politicians, scammy businessmen, smooth-talking romantic partners - would be able to argue her into anything. And so she takes the obvious and correct defensive manuever - she will never let anyone convince her of any belief that sounds "weird"
Bostrom's simulation argument, the anthropic doomsday argument, Pascal's Mugging - I've never heard anyone give a coherent argument against any of these, but I've also never met anyone who fully accepts them and lives life according to their implications.
People used to talk about how terrorists must be very poor and uneducated to fall for militant Islam, and then someone did a study and found that they were disproportionately well-off, college educated people (many were engineers). ... a sufficiently smart engineer has never been burned by arguments above his skill level before, has never had any reason to develop epistemic learned helplessness. If Osama comes up to him with a really good argument for terrorism, he thinks "Oh, there's a good argument for terrorism. I guess I should become a terrorist," as opposed to "Arguments? You can prove anything with arguments. I'll just stay right here and not do something that will get me ostracized and probably killed."
u/xamueljones [+3] My arch-enemy is entropy (a day later)
Wow, this sounds like Bayesian reasoning for why people would follow Majoritarianism.
It basically boils down to the idea that on subjects where you are uneducated or have reason to believe that you are more ignorant about than the majority of people, you should follow what the crowd does or at least until you take the time to learn enough to perform better or know more than the majority.
I personally use this heuristic sometimes with the exception of major political issues, since that's a topic where there often is no clear majority.
u/notmy2ndopinion [+3] Concent of Saunt Edhar (2 days later)
I struggled to understand this concept initially because a) the word epistemic tends to trip me up, vocab-wise and b) the idea of epistemic learned helplessness is so foreign to me now, since almost everything I deal with in my daily profession deals with ambiguity.
However, my fiancée used to tell me that she "dislikes all politics" since she saw equal value in the arguments on both sides of the aisle that she decided to take no position. Now that she's in Social Work school, her views are more nuanced and informed rather than shrugging helplessly at everything political, so to speak.
u/MugaSofer [+3] (14 hours later)
Scientists have a bad reputation because of horrific experiments in the past (think of Twig but more medieval... or the Duke of Gibea in the Kingkiller chronicles) as well as the creation of several existential threats like an unfriendly magical AI, grey goo, nuclear weapons, and bioterrorism.
Sounds like a reasonable reason to ensure nobody else is doing it to me. I'd rather the planet wasn't destroyed, thanks.
Sure, you can have some people doing experiments under carefully controlled conditions, but out in the wild? That would be ... actually irrational.
u/notmy2ndopinion [+1] Concent of Saunt Edhar (a day later)
Almost sounds like a good reason to lock them up in concents (Anathem)... or put chains around their necks (maesters)
u/CFCrispyBacon [+2] (an hour later)
What if they're concerned that developing these extrasensory abilities will lead to another rise and collapse of another kind of magic? These abilities sound like more different magic, and the only way to survive is obviously to get rid of everything that is magical, that works on unproven methods. The only things that could possibly be safe are proven methods that have worked for generations, and we will use force if necessary to save civilization from the unsafe path you are leading it on, etc.
u/notmy2ndopinion [+2] Concent of Saunt Edhar (a day later)
I have spent a lot of time (TOO MUCH TIME) researching about how realistic it would be to have something like darkvision/infravision from D&D and actually trying to put together the type of sensory organs it would take (eyes aren't the right size, the lens doesn't refract UV or infrared light correctly; the skin would be the organ most easily adapted to detect heat since it already does to some degree)... then pulling it back into fantasy (orcs would be the best race to have this ability), thinking about the implications (so they'd be great at tracking blood and bodily fluids, highly sensitive to heat fluctuations, need a lot of skin exposure to be effective, etc...)
So in a nutshell, I was trying to rationalize all of these ESP abilities but I was confusing realism with magic. It certainly is much easier to hand wave it as 'new magic' but that makes it much less fun when it is such a black box in a setting intended to be scientific, rational, and explore biology and pathophysiology.
Of course, all of my background characters would just call it magic and be done with it.
u/CFCrispyBacon [+1] (a day later)
And all your antagonists will be afraid of it, and there's your story. Getting rid of something because it "feels wrong", or leaping to conclusions about what appear (on the surface) to be related effects are perfect for your antagonists-reasonable conclusions, but not rational ones.
u/NotAHeroYet [+2] City of Angles Municipal Government (2 hours later)
They could favor keeping what they have, over reaching for the stars.
They could believe, perhaps correctly and perhaps not, that all magic comes from gods- plural, not singular- that using these magics is a form of worship and, simultaniously, makes these gods stronger. They could be worried because unlike their, safe gods and magics, which have almost never destroyed stuff that they didn't mean to, these have horible extremes out there, and you shouldn't let it reach its true potential.
They could believe magic creates magic for other reasons, too.
Magic could legitimately be dangerous, if used haphazardly, and they make plans to minimize the threat with kill-on-sight. perfectly valid, especially if most mages didn't ever teach one another or use sufficient safety levels.
Aiming for large groups of mages may be foolish, but they think the potential for harm is greater, but the likelyhood is just as high, rather than assuming that they learn safety from one another's mistakes.
Meanwhile, mages tend to get vengeful, if they last long enough to survive, (attempted murder does that to a person's outlook) and therefore are even more dangerous in groups. they can't let one mage go, because a single mage is a bomb waiting to go off, but they can't defuse them without making enemies and making groups of mages dangerous as well.
In other words, they're dealing with superhumans who have every reason to be angry with them, but who also very likely could accidentally wreak havoc if they aren't killed. There is no third option that isn't inferior, at least not that they can see. Kill-on-sight or let them wreak havoc.
u/xamueljones [+2] My arch-enemy is entropy (3 hours later)
I'm curious, how far along are you with this book? Because it sounds really ambitious and I remember you posting a few times about it starting from a year ago.
I wish you luck with it!
u/notmy2ndopinion [+2] Concent of Saunt Edhar (a day later)
Thanks! I've got the structure vaguely outlined and the main characters still need mini-arcs. I have a lot of the medical cases, fantasy-fantastic examples, and rationality principles all bubbling around in my brain. The major shift for me is that I overhauled the time for the setting -- originally I was going to set it in the traditional D&D 'pseudo-medieval' time, but it didn't feel right since I wanted the medical backdrop to be more than leeches and bloodletting.
It all changed a few weeks ago when I read about medicine in the 1800s in a book called "Dr Mutter's Marvels" (of the Mutter Museum fame with weird medical monstrosities in Philly). So I feel ready, esp now that it's NaNoWriMo, to start writing in earnest!
u/LiteralHeadCannon [+2] (3 hours later)
Societal/institutional inertia is a hell of a thing.
u/ArgentStonecutter [+2] Emergency Mustelid Hologram (8 hours later)
If they can rationally project the dangers of industrial civilization that has, in fact, already demonstrated "knowledge that could destroy the world when it was in the hands of the wrong people" are you sure they are not rational? Rational doesn't mean correct. You can only make decisions based on the information available to you.