103 KiB
[RST] Pokemon: The Origin of Species, Ch 50 - Comfort Zone Expansion
-
Author: u/DaystarEld Pokémon Professor*
-
URL: https://www.fanfiction.net/s/9794740/50/Pokemon-The-Origin-of-Species
-
Score: 66
-
Created: 2017-12-01T12:00:54
Post:
Comments:
u/DaystarEld [+16] Pokémon Professor (a minute later)
Welcome back everyone! Quick disclaimer/warning: this chapter contains a section with math. I apologize in advance, but if my characters are going to start using Bayes' Theorem while calculating things, I figured it was worth actually showing them going through it at least once for anyone interested in learning. If your eyes glaze over and you find it horribly tedious, I totally understand, and apologize for not finding a more entertaining way to include it.
That said, I do think it's an important bit of math to learn for any aspiring rationalists, and there are plenty of resources online that can do a better job teaching it than I can, including the one you can find here. Whether you take the challenge on or not, I hope you enjoy the chapter, and all feedback welcome, as always! (Particularly if you think I got something wrong or explained something poorly. I know I say that a lot, but I super extra mean it this time.)
u/gbear605 [+15] history’s greatest story (13 hours later)
Knows better than Mrs. Verres? Who's spent years in the field, been through so much more? What are the odds, of that? What's the prior, that a 12 year old who just started in a vocation would have better instincts, better insights? What are the sheerly lopsided odds against it?
Something something modest epistemology?
u/DaystarEld [+7] Pokémon Professor (2 days later)
Yep. Generally thinking that you have a better understanding of a situation than an expert requires something like demonstrable mastery or understanding of why the expert is wrong. In this case, Leaf just wanted to be right, which is not sufficient when facing an expert telling you you're wrong.
I like EY's take on it overall.
u/The_Magus_199 [+12] Ankh-Morpork City Watch (8 hours later)
Holy shit, that was ominous. I was expecting a fun chapter about Red’s experiences in this gym, and instead we get Leaf’s life being slowly consumed by her attempts to uncover a dangerous conspiracy only to find out that Laura’s in trouble...
(Also, I note that Leaf didn’t consider whether Giovanni could have calculated what he said to point her to his enemies, regardless of if they actually did it. :p)
u/DaystarEld [+8] Pokémon Professor (9 hours later)
(Also, I note that Leaf didn’t consider whether Giovanni could have calculated what he said to point her to his enemies, regardless of if they actually did it. :p)
I mean, he just has so many enemies, what can he say, you're bound to find some if you go poking around at any given conspiracy...
u/nipplelightpride [+2] (3 days later)
This chapter gave me the bad feeling that something is going to happen to Laura and Leaf is going to have to rediscover whatever Laura did.
u/None [+12] (3 hours later)
[deleted]
u/Trips-Over-Tail [+10] Death of Crabs (5 hours later)
MissingNo.
u/DaystarEld [+3] Pokémon Professor (9 hours later)
The fact that tier 3 threats were almost completed is rather annoying
Almost completed?
They're not exclusively related to the stormbringers, but what else can cause a tier 3 threat?
On a rare occasion, if for example there's a whole group of Tyranitar on a rampage instead of just one, the threat would quickly approach that of a Stormbringer. Those are pretty rare, however, and far less frequent than the already somewhat infrequent Stormbringer flights over populated areas (which happen roughly once a year), so those are the vast majority.
What is the true "top" of the threat range?
What's the greatest threat you can think of? :)
u/Cariyaga [+6] Kyubey did nothing wrong (13 hours later)
An angry, intelligent legendary? Like, say, Mewtwo.
u/sidhe3141 [+13] (15 hours later)
The Champion going Renegade.
Massive ecosystem disruption from the Fairy type reappearing.
Creator legendaries waking up and fighting.
One creator legendary waking up without the opposite (explicitly an x-threat in canon).
An intelligent and contagious Glitch manifestation.
Arceus deciding to go Genesis 6-8 on the world.
u/Trips-Over-Tail [+4] Death of Crabs (a day later)
Surfing up and down Cinnabar Island?
u/LeonCross [+4] (a day later)
Depending on what you've adjusted from canon, there are a ton of legendary ed that absolutely shit on the trio in terms of power level. Abstracts, the creation trio, arceus, etc.
u/FireHawkDelta [+9] ** (3 hours later)
The math was actually pretty exciting! Seeing Bayes Theorum in practice helped me understand it more than just the idea of it.
u/DaystarEld [+6] Pokémon Professor (3 hours later)
Glad to hear it!
u/sidhe3141 [+9] (16 hours later)
Something that had been floating around my head for a while:
I suppose it's been too long since I saw this place through fresh eyes.
Maybe I had been turning away Mastery challenges to keep myself from seeing it. On some level, maybe I thought I'd lose my nerve if I was reminded. It's certainly making me rethink all of it now.
But what's done is done. What happened happened, and there's no unbaking this cake.
There is one last formality before you can call yourself a Master, Blue. Three questions, all with one answer. You don't have to tell me; they'll tell you inside. But I think you can work it out, if your history classes didn't cover it.
First, why does the region have a Champion rather than a General?
Second, why did we walk to the Plateau instead of flying?
Third, why is such a desolate place called Victory Road?
u/DaystarEld [+8] Pokémon Professor (17 hours later)
That's pretty good :) Is that from something, or something you've been imagining?
u/LazarusRises [+2] (a day later)
What's this from? Don't think I get it.
u/sidhe3141 [+2] (a day later)
It was something I was thinking might be close to the end of the story.
u/LazarusRises [+2] (a day later)
Can you explain it? I feel like I'm missing the answer to a riddle.
u/sidhe3141 [+2] (14 days later)
The idea is that it's called Victory Road ironically: "when wars are fought with Pokemon, this is what victory looks like".
u/None [+10] (2 days later)
I think this is literally the first time I've seen a character get warned that something is too dangerous and actually take that advice seriously. Big increase in my respect for Leaf as a character and you as an author. Its nice to have it demonstrated that taking agency doesn't mean being an idiot
u/DaystarEld [+8] Pokémon Professor (2 days later)
It's super rare in most fiction because danger = conflict = the plot for pretty much every story where someone is warned that something is dangerous. Protagonists get into extraordinary danger and survive because they have Plot Armor, by and large: to show a Protagonist who's able to differentiate between danger they can handle and danger they can't, they have to actually once in awhile hold back out of legitimate and justified fear of it.
Harry in HPMOR is smart enough to know not to look for the Chamber of Secrets when McGonagall tells him that it led to a student's death years ago, but even then once he learned what potential powers lay waiting inside for him he was tempted to go looking for it. In general he's pretty conscientious about safety, but we're all fallible to desires that can overcome justified caution.
u/DaystarEld [+5] Pokémon Professor (a minute later)
Typo/math correction thread!
u/None [+5] (2 hours later)
[deleted]
u/DaystarEld [+3] Pokémon Professor (2 hours later)
I did indeed :) Thanks!
u/MultipartiteMind [+6] (3 hours later)
"Sometimes an incident that looks like a Tier 1 ends up being a Tier 2, while other times a Tier 2 threat is misidentified as a Tier 1"
These are the same situation; based on the reply, should the first one be 'that look like a TIer 2 ends up being a Tier 1', so that that first one is the one where resources are overcommitted?.
"Who can tell me why that's a problem?"
"Because the threat assessment keeps us from overcommitting resources on one end, or not committing enough on the other"
Being kept from overcommitting resources sounds like a good thing; should this be 'leads to us overcommitting resources'?
Curiously, the two cases are sort-of consistent in their parallel.
"It could look small and be big, or else it might be big and look small! Why is that bad!?"
"Sir! We might not use too many resources, or we might use too few resources!"
u/DaystarEld [+3] Pokémon Professor (3 hours later)
Curiously, the two cases are sort-of consistent in their parallel.
Right, they're both meant to be a good thing :) "keeps us from overcommitting" and "keeps us from not committing enough." It's a bit awkwardly stated though, so I'll edit it, and the Tier 1/2 misidentification.
Thanks!
u/MultipartiteMind [+3] (4 hours later)
Ahh, I see! (I took 'the threat assessment' to mean 'the mistaken threat assessment'... maybe 'is in order to' or 'is intended to' or 'is supposed to', and/or 'make sure'->'stop us from (overcommitting...)'... clearer that Red is talking about the purpose of a threat assessment, rather than the consequences of a mistaken threat assessment... *nods*)
u/masasin [+3] (6 hours later)
Those are false positives (or false negatives, depending how you look at it).
u/MultipartiteMind [+2] (2 days later)
Yes, they are.
(If you view Tier 1 as more normal/safe/non-alarming than a Tier 2, then perhaps treating the Tier-1-thought-to-be-Tier-2 as false positives, Tier-2-thought-to-be-Tier-1 as false negatives.)
u/KnickersInAKnit [+4] (3 hours later)
Strong suit not strong suite
u/DaystarEld [+3] Pokémon Professor (3 hours later)
Fixed, thanks!
u/Hermaan [+3] (an hour later)
"thoughts already Ryback's message" -> "thoughts already on Ryback's message"?
u/DaystarEld [+3] Pokémon Professor (2 hours later)
Fixed, thanks!
u/Makin- [+3] homestuck ratfic, you can do it (an hour later)
"weighing tha" -> that
(right after the above) "TIer" -> Tier
u/DaystarEld [+3] Pokémon Professor (2 hours later)
Fixed both, thank you!
u/GriffinJ [+3] (2 hours later)
When you're listing out the guidelines for Tier 1/2 determination, you might want to clarify 5 so that it's clear that you're referring back to the pokemon in step 3, unless I'm misreading things.
u/DaystarEld [+5] Pokémon Professor (2 hours later)
Edited to emphasize the THEY, hopefully that makes it more clear :)
u/GriffinJ [+3] (3 hours later)
That works :)
u/masasin [+3] (6 hours later)
As a final result, leaf calculates
0.0852/(0.0852 + 0.2112) = 0.287, but reports 0.2677. Did I miscalculate?
u/DaystarEld [+3] Pokémon Professor (8 hours later)
Nope, that's just me changing a variable and forgetting to update the answer like a dumb :D Thanks!
u/thrawnca [+3] Carbon-based biped (a day later)
36% of incidents in Kanto are tier 1
Should be tier 2.
u/DaystarEld [+3] Pokémon Professor (a day later)
Fixed, thanks!
u/Trips-Over-Tail [+3] Death of Crabs (a day later)
This is old, but I was rereading through the whole story, and in Chapter 25 Leaf writes "Species of pokemon that have not existed for millennium are returning to the world."
The plural of millennium is millennia. If it is leaf's mistake, Laura should correct it.
u/DaystarEld [+3] Pokémon Professor (a day later)
Fixed, thanks!
u/Malakbel [+3] (a day later)
"Then please, please don't react like I'm afraid you will to what I'm about to say. Please trust that I have good reasons for it."
please don't react like I'm afraid you will to what I'm about to say
After rereading it a bunch of times I finally get what Mrs. Verres is saying here, being afraid of how she expects Leaf to likely react. But it still reads really weird after I finally got it.
u/DaystarEld [+3] Pokémon Professor (a day later)
I'll try to make it more clear, thanks!
u/Malakbel [+3] (2 days later)
=)
u/Hermaan [+4] (an hour later)
I really loved this chapter, I have to admit skipping the math part for now though. The ending already got me hyped for what's to come.
u/DaystarEld [+6] Pokémon Professor (3 hours later)
Glad you enjoyed it :) I'm often nervous about Leaf-heavy chapters, and this is the second full-Leaf-perspective chapter so far, so it's good to hear it received well!
u/Hermaan [+10] (3 hours later)
I think they are a nice change from the more action-heavy Blue chapters. While I think the Red chapters have a nice mix between the occasional action and him theorizing and studying, it's interesting how Leaf faces problems, that are in some ways similar to Red's struggles, but approaches them in her own way.
With Red it often seems like he perceives looking for Oak's help as some kind if weakness, because he fears getting extra passes, because of starting off working in the lab. I feel like Leaf looks up to Laura and values her as a mentor, while still trying to rise on her own merits.
u/None [+3] (2 days later)
I'm often nervous about Leaf-heavy chapters,
They are honestly one of my favorite parts, since the social side of affecting the world is so often underexplored in rat fiction
u/DaystarEld [+2] Pokémon Professor (2 days later)
\o/!
u/Trips-Over-Tail [+3] Death of Crabs (3 hours later)
Let's hope Leaf never gets a Vileplume, or she'll be drawing blanks trying to name it.
u/DaystarEld [+5] Pokémon Professor (8 hours later)
Venusaur: "This pokebelt ain't big enough for TWO grass/poison types with giant pink and white spotted flowers on top!"
u/Trips-Over-Tail [+10] Death of Crabs (9 hours later)
"...Whose name begins with V!"
The posturing would actually work, because of the two only Venusaur can learn Roar.
Not that anyone in Pokemon player history has ever allotted a Venusaur move slot to Roar, but still.
I don't know if you've decided on a nickname for Pichu yet (if she'll even get one), but I've always liked Amber for female electric types, given the role of amber in early science and the subsequent origin of the word "electricity".
u/DaystarEld [+6] Pokémon Professor (9 hours later)
Thanks for the suggestion :)
u/XxChronOblivionxX [+4] (8 hours later)
"Oh, I should introduce everyone… this is Glen, that's Chron," the boy to Red's other side raises his hand.
Ayy! Bit of a weird first name, but I can dig it.
u/DaystarEld [+2] Pokémon Professor (9 hours later)
I can update it if you have a preferred different one, just figured you'd gone to sleep like a reasonable person by the time I decided to add it :)
u/XxChronOblivionxX [+2] (9 hours later)
I certainly would have fallen asleep by then if I were better at making decisions.
But Kron is an actual name, would probably work better.
u/DaystarEld [+2] Pokémon Professor (9 hours later)
Yeah, I figured it was just a regional spelling difference that made a fun portmanteau :)
u/masasin [+4] (4 days later)
/u/DaystarEld, /u/daydev, /u/Iijil
I just remembered that we could have used the odds ratios instead of all the complicated math.
- prior odds * relative likelihoods = posterior odds
- relative likelihoods = posterior odds / prior odds
We know that:
- Posterior odds of R1 are 79:21
- Prior odds (T1:T2) are 64:36 So the relative likelihoods (R1 | T1:T2) are 79/64:21/36
We can use that directly in the next step, where the prior odds are 2:15.
79 : 21 ÷ 64 : 36 × 2 : 15 ---------- 79 : 280So P(T1 | R1) = 79/(79+280) = 22.01%.
Doing it for R2:
33 : 67 ÷ 64 : 36 × 2 : 15 ---------------- 33/32 : 335/12So P(T1 | R2) = (33/32)/(33/32+335/12) = 3.56%.
Would you prefer something like this? Maybe when Leaf is showing Red the right way to do it, since he's already trying to use odds instead of probabilities. It wouldn't take too much longer, and maybe in fact be shorter.
edit:
That way, you'd also easily be able to see how the odds change every time you get a certain kind of report.
The relative likelihood of R1 | T1:T2 is 2.116071429:1, and O(R2 | T1:T2) is 1:3.609427609. Call them 2.116:1 and 1:3.609.
Let's say it's Tyranitar again, and we receive six R1 and two R2. The order doesn't matter.
2 : 15 2.116 : 1 2.116 : 1 2.116 : 1 2.116 : 1 2.116 : 1 2.116 : 1 1 : 3.609 1 : 3.609If you want to shorten it, it's 2 * 2.116^6 : 15 * 3.609^2 . The posterior odds are 179.525 : 195.373, so P(T1|6 R1 and 2 R2) = 47.89%. (It would have been 76.83% with just one R2, and 92.3% with none.)
edit 2: log likelihoods
Because you add likelihoods instead of multiply, and you have a single number, the Rangers would probably use this method in real life. 1 deciban is 0.1 log10 likelihood. 1 dban is also a deciBel (dB).
Log likelihood of T1 is log(2/15) = -0.875 = -0.875 ban = -8.75 dB Log likelihood of R1 is log(2.116) = 3.26 dB (You can also find this by log(79/21) - log(64/36).) Log likelihood of R2 is log(1/3.609) = -5.57 dB
When you get 6 R1 and 2 R2, you add 6 L(R1) and 2 L(R2) to L(T1). You end up with -0.33 dB. (10^-0.033 / (10^-0.033 + 1) = 0.48, which we had before.)
dB: evidence strength (Using Kass and Raftery) 0 to 5: weak evidence 5 to 14: positive evidence 14 to 22: strong evidence 22+: very strong evidenceSo, -0.33 dB is weak evidence for T2. We'd need 7 reports of R1 with no R2 to have strong evidence, but we can consider it moderate evidence after just 5 R1 0 R2. On the other hand, if we got a single R2, we would already have strong evidence. Another two, and we'd get to very strong evidence.
Bonus: People don't tend to see differences in probability until that difference is about a deciban (e.g., 50 to 55.7%, or 99% to 99.2%).
u/DaystarEld [+2] Pokémon Professor (11 days later)
I'll definitely have Red end up learning to do it that way, since it's much less math intensive. Thanks!
u/Revisional_Sin [+1] (10 days later)
Sweet Arceus, odds ratio is so much better!
Edit: I couldn't quite understand it from masasin's comment, this helped: https://betterexplained.com/articles/understanding-bayes-theorem-with-ratios/
u/masasin [+2] (10 days later)
Yup yup.
u/Revisional_Sin [+1] (10 days later)
/u/DaystarEld
Good effort, but that sequence made me run screaming from Bayes, and I like maths!
My humble suggestion: have Leaf quizzing Red on how you could possibly use Bayesian probability, given how laborious the calculations are.
She goes ahead and calculates it the original way, then Red excitedly shows her the odds ratio method.
You could keep Red being bad at maths, and have her save face, by having him comment that he'd always mess it up when he tried to do it her way.
u/masasin [+2] (10 days later)
/u/DaystarEld
Yeah, something like "Why didn't you just use odd ratios?" And Leaf facepalms at the obvious oversight (I did).
u/DaystarEld [+2] Pokémon Professor (11 days later)
Good idea, but I want to keep the Group method too as a way to show practically what it means in real terms. I think I'll include /u/masasin's odds ratio method in the next chapter as the way Red finally learns to do it with a minimum amount of math :)
u/masasin [+1] (11 days later)
Oo, looking forward to that.
By the way, I'm moving to Belgium next month, so if you want to chat on Discord etc, the sooner the better.
u/DaystarEld [+1] Pokémon Professor (11 days later)
Heh I just sent you a comment on this, Discord works too, I'm Daystar Eld#4161 there.
u/Revisional_Sin [+1] (11 days later)
In that case, I think there are some ways you can still make Leaf's actions clearer.
I'd have her sketch a probability tree or two, in order to illustrate the problem more intuitively.
She also jumps into a sequence of calculations that magically end in the answer, I'd reorder it slightly:
Have her start off by looking at Bayes theorem, and musing: "Okay, I need x, y and z... I've got y and z, so I need to calculate x somehow. To get x I'll do this..."
u/DaystarEld [+2] Pokémon Professor (12 days later)
Oh yeah, I'd definitely use the tree if fanfiction let me draw it out. Might still be worth describing it...
u/Iijil [+3] (3 hours later)
I have a question about the math.
One of the assumptions we start with is that Tier 1 reports over all pokémon have a 21% chance of actually being Tier 2. Why exactly can't we just take that as our final answer?
I guess that number is too far away from what we expect the odds to be so we assume there is something special about reports about Tyranitars in particular.
Instead we calculate that over all pokémon Tier 1s are reported accurately with 71% chance and Tier 2s are reported accurately with 76% chance.
We then continue to use those numbers as the probability that a Tyranitar Tier 1/2 event is reported accurately.
Why is it any more reasonable to restrict to Tyranitars in that context?
u/FeluriansCloak [+8] (3 hours later)
Because if we have additional information, we should update our thought process based on it. We know that tyranitar attacks are more likely to actually be tier 2, so we should consider that when doing a threat assessment.
One of the places this shows up a lot in real life is in medical tests. If a test for disease A has a 10% false positive rate, you might naively think that if you test positive there's a 90% chance you're sick. But that ignores the fact that most medical conditions have a relatively low incidence rate. If disease A is typically found in just 1% of the population, then it means that most of the positives were actually false positives, since it was more likely to start with that you weren't sick. (This is the idea of "priors" sometimes discussed). Using the hypothetical 100 person population, and ignoring false negatives, for this test you would expect 11 total positive test results, only one of which corresponded to an actual positive.
u/Iijil [+3] (4 hours later)
The difference between the disease example and the Tyranitar version is that with the diseases the accuracy of the test is given as it applies to the specific disease we are talking about.
For Tyranitars we accuracy of reporting is derived from the statistics we gathered about all pokémon events. That is like saying medical tests in general have a 10% false positive rate, so we should apply that to this disease as well.
So if we have no data about the specific test how can we get the probabilities that we need to apply bayes?
What is the reasoning for keeping specific probabilities fixed when going between general case and specific case? You can get vastly different results for different choices on what to keep fixed.
u/FeluriansCloak [+4] (5 hours later)
This is absolutely a fair point, and is an assumption we need to make for the example in the story. I think the idea is that given no other information, that's the best we have to go off of.
u/Iijil [+3] (6 hours later)
So we take the rate of error given a report, convert that into the rate of error given an actual classification, assume that this rate is the same when only looking at Tyranitars, and convert back into the rate of error given a report. Resulting in the 26.77% Leaf arrives at.
Alternatively we can take the error rate given a report and assume that is the same when looking only at Tyranitars. We get a 79% chance.
How do we decide that we are better off doing it one way or the other?
Personally I would take Leaf's previous comment about being surprised by the actual classification and assume that reports about Tyranitars are hard to get right. I'd mostly go by the 2:15 Tyranitar odds and not give the report a lot of weight. So Leafs number makes more sense to me. But I don't understand how or if it is mathematically more justified than the other approach.
u/DaystarEld [+3] Pokémon Professor (8 hours later)
If I understand your point correctly, yes, there can absolutely be a more accurate number found if you look only at Tyranitar reports and use that to adjust the 2/17. They just don't have that information in front of them now.
u/daydev [+2] (9 hours later)
Oh, I understand now what I missed! The stats for Tier 1/Tier 2 reporting accuracy are for all Pokemon and Tyranitar have unusually high proportion of Tier 2. I somehow assumed the accuracy percentages were for Tyranitar reports and was very confused.
u/DaystarEld [+2] Pokémon Professor (9 hours later)
Ah, yes, that would be a bit redundant :)
u/Iijil [+2] (9 hours later)
Yes, data about only Tyranitars would be preferable, but in the absence of that data why do they estimate it in the specific way they do?
If they had the data that 20% of reported Tier 1 Tyranitar rampages are actually Tier 2, they wouldn't need to use bayes anymore, because that statistic is exactly what they are looking for. The high likelihood of Tyranitars being Tier 2 would be automatically considered during data collection. We would have very few Tier 1 Tyranitars being reported in the first place, but once we encounter that situation we go to the statistic we have.
So in the situation where they have the statistic that 21% of reported Tier 1s are actually Tier 2, why is it not justified to assume that will hold for Tyranitars?
And if we think Tyranitars are different, then why, after figuring out the reporting error rates for given actual classification, is it justified that those error rates will be the same for Tyranitars?
Ahh, I think I got it while writing this post. If the world suddenly changed to a world where the ratio of Tier 1 to Tier 2 is 2 to 15 instead of 36 to 64 it would make sense for reporting errors for a given classification to stay constant, but not for reporting errors for a given report. So it would be correct to treat the change to Tyranitars like that as well. Am I making sense with that?
u/DaystarEld [+2] Pokémon Professor (9 hours later)
I think so :) The way I see it, yeah, there could suddenly be a bunch of Tyranitar attacks in the next couple years that massively change the rate of expected Tier 1 vs Tier 2, but people wouldn't necessarily get better at recognizing it right away. Or people might get worse or better at reporting the events from one decade to the next, and those two different factors will be important to determining how to treat a report.
u/daydev [+2] (4 hours later)
It seems to me that there's a slight but important difference between the medical test example and this one. The false positive for medical test supposes that out of 100 people who don't have it, 10 will test positive. This example gives it the other way around, out of 100 positive results, 79 actually have it, this seems like the actual answer. It's possible I don't understand Bayes well enough, but it seems to me it should be written the other way around "Tier 1 incidents are reported as Tier 2 21% of the time".
UPD: Don't mind me, I'm just stupid, I didn't realize the report percentages were for all Pokemon incidents, not just Tyranitar incidents.
u/chaos-engine [+3] (9 hours later)
Woah, tons of suspense building up here!
These rational fics have the main characters acting so smart that I forgot these are a bunch of 12 year olds. In my mind I keep thinking of everyone as being in their late teens :)
Having a blast regardless though.
u/DaystarEld [+4] Pokémon Professor (9 hours later)
Glad to hear you're enjoying it! And yeah, they're not typical 12 year olds, but I've worked with a couple who are pretty close :)
u/None [+3] (2 days later)
I like the occasional reminders, makes it feel more grounded than things like HPMOR where you basically need to ignore the characters notional ages
u/masasin [+2] (7 hours later)
I'm stuck and/or confused.
The terminology I use is:
T1, T2 = Actually Tier 1/2 R1, R2 = Reported Tier 1/2
I get this part:
Givens P(T1) P(T2) Prior (an attack occurred) 0.12 0.88 R1 0.79 0.21 R2 0.33 0.67 What we're looking for is P(T1 | R1), the probability that a Tier 1 Report is actually a Tier 1.
What Leaf tries to find is the probability of Tier 1 being reported accurately, which, to me, would be P(R1 | T1) / P(T1). We don't know P(R1 | T1), so we have to use Bayes's rule:
P(R1 | T1) = P(T1 | R1) * P(R1) / P(T1)
But we don't know P(R1) either.
What Leaf instead does is calculate the percentage of Tier 1 reports that actually represent Tier 1, which we can do:
P(R1 | T1) / P(R1) = P(T1 | R1) * P(R1) / (P(T1) * P(R1))
which cancels out to P(T1 | R1) / P(T1).
However, she calculates P(T1) as:
P(T1) = P(T1 | R1) + P(T1 | R2)
instead of:
P(T1) = P(T1 | R1) * P(R1) + P(T1 | R2) * P(R2)
And this got me completely stuck. Help?
u/DaystarEld [+2] Pokémon Professor (9 hours later)
Can you help explain why
P(T1) = P(T1 | R1) * P(R1) + P(T1 | R2) * P(R2)
Is the better formula for the rate of T1? I may be having trouble following the format, which is totally standard and the one I should be familiar with, but am still trying to get the hang of :)
u/masasin [+2] (9 hours later)
If you know P(T1 | R1) (how often reports of Tier 1 are actually Tier 1) and P(T1|R2) (how often reports of Tier 2 are actually Tier 1), you still need to know the individual frequencies of R1 and R2.
As an extreme example, imagine there were ten thousand attacks. 9900 (99%) were reported as Tier 1, and 100 (1%) were reported as Tier 2. Using the percentages from this chapter, you'd end up with 79% of Tier 1 reports actually being Tier 1 (7821), 21% of Tier 1 reports actually being Tier 2 (2079), 33% of Tier 2 reports actually being Tier 1 (33), and 67% of Tier 2 reports being Tier 2 (67).
In total, you have ten thousand attacks, 7854 (78.54%) of which were Tier 1, and 2146 (21.46%) of which were Tier 2.
If you want to calculate it without P(R1) and P(R2) (99% and 1% respectively, in this example), you would end up with:
- P(T1 | R1) = 0.79
- P(T1 | R2) = 0.33
P(T1) would then be equal to 79% + 33% = 112%. If you use the rate at which R1 and R2 occurs, you'd have:
P(T1) = 0.79 * 0.99 + 0.33 * 0.01 = 0.7854, or 7854 out of 10000, which is exactly what we had.
edit:
What Leaf and Red were looking for was P(T1 | R1), which is the probability that a Tier 1 attack occured, given a report of a Tier 1 attack. In this toy example, it would be 79%. But you do not have R1 and R2.
u/DaystarEld [+2] Pokémon Professor (11 hours later)
If you know P(T1 | R1) (how often reports of Tier 1 are actually Tier 1) and P(T1|R2) (how often reports of Tier 2 are actually Tier 1), you still need to know the individual frequencies of R1 and R2.
I'm not sure I follow why you need to know the frequencies. Isn't it enough to know what % of them are accurate, regardless of how frequently each one is reported?
The point of this section:
100 Events reported as Tier 1
79 are actually Tier 1
21 are actually Tier 2
100 Events reported as Tier 2
67 are tier 2
33 are Tier 1
Is to essentially give that information as a hypothetical, since they don't have the actual frequencies of T1 vs T2 reports. Since their goal is to just figure out, as you say, whether this specific T1 report is in fact accurate, I'm a little confused as to why it's necessary to know how often T1 reports occur at all, rather than just how often they're accurate.
This:
P(T1 | R1) = 0.79
P(T1 | R2) = 0.33
P(T1) would then be equal to 79% + 33% = 112%.
Seems to be answering how frequent T1 is compared to T2 for general pokemon reports, but we already have that answer for Tyranitar reports: 12%.
u/masasin [+2] (11 hours later)
Did you see my second comment/edit? I think your wording might have been a bit off.
we already have that answer for Tyranitar reports: 12%.
That's exactly it.
I'm a little confused as to why it's necessary to know how often T1 occur at all, rather than just how often they're accurate.
If you don't know how often T1 occurs, then the first part (12% vs 88%) gives you absolutely zero information. You'd be working with likelihoods and ignoring your priors.
You're looking for P(T1 | R1), or the probability that a Tier 1 incident occurred given that a Tier 1 incident was report, and you would need to have your answer be 0.79. You could also ask what percentage of Tier 1 reports are actually from Type 1 incidents:
P(R1 | T1) / P(R1) = P(R1 | T1) / (P(R1 | T1)*P(T1) + P(R1 | T2)*P(T2)) = P(T1 | R1) / P(T1). With P(T1) offering no additional information, you default back to P(T1 | R1) = 0.79.
u/DaystarEld [+2] Pokémon Professor (11 hours later)
I just did, sorry, I had the tab open for awhile so I just answered that one before refreshing and seeing the new one.
If you don't know how often T1 occurs, then the first part (12% vs 88%) gives you absolutely zero information. You'd be working with likelihoods and ignoring your priors.
But the 12% is the prior? I don't get what you mean by it gives you zero information.
Since they're specifically looking at a Tyranitar report, the amount of Tyranitar events that have been T1 or T2 in the past is far more accurate a prior than the total amount of T1 vs T2, of which Tyranitar is just a subset, no?
That's my understanding of it after being told that there was no reason to apply the frequency of T1 events to the frequency of Tyranitar T1 events, anyway. It sounds like you're saying there is actually a reason to do that?
u/masasin [+2] (11 hours later)
If you don't know how often T1 occurs
^ Should have been "If you don't care how often T1 occurs" (i.e., if you ignore T1)
You're right. The 12% (T1) is your prior, but you're not using it when you're calculating P(R), which is dependent on P(T1) and P(T2). By T1 I mean specifically the Tyranitar Tier 1 events.
No matter which way you look at it, unless you take the straight P(T1 | R1) = 0.79 (which is the likelihood), you have to use P(T1). It's unavoidable.
Now, if you had actually meant the 0.79 to be P(R1 | T1) (the percentage of Tier 1 incidents reported as Tier 1) rather than P(T1 | R1) (the percentage of Tier 1 reports that are actually Tier 1 incidents), everything goes much more smoothly, and it's a simple step to get P(T1 | R1) after that.
u/DaystarEld [+2] Pokémon Professor (18 hours later)
Okay, so you're basically saying that because I'm mixing my reports (probability of a Tyranitar event being Tier 1 or Tier 2 vs probability of general incidents being T1 or T2) I'm skipping a step in figuring out what the actual relationship is?
If I was using general incident frequency and general report accuracy, that would be fine.
If I'm using a specific pokemon incident frequency, and specific pokemon report frequency, that would be fine.
But mixing both means the relationship isn't as clear cut and the actual number of general reports matters to how much confidence I should be giving the smaller subset of Tyranitar T1 or T2 reports.
Is that about right?
u/masasin [+2] (a day later)
Nope. Do you want to meet on Hangouts or Skype or something? It may be easier to explain.
I'll avoid using terms like accuracy for now, and use the "standard" notation. I was talking about Tyranitar events being T1 or T2, but that does not matter in the larger scheme of things. If general attacks are 60:40, then you have P(T1) and P(T2) as 0.6 and 0.4 respectively, and you just adjust your calculations accordingly.
The issue is that you have P(T1 | R1), the percentage of Tier 1 reports that are actually Tier 1. This was actually what you were looking for though (how to respond if you get a Tier 1 report), and would have been the answer. It is independent of the percentage of incidents that are Tier 1.
In your case, P(T1 | R1) was 79%. That is, if you receive a Tier 1 report, it is actually Tier 1 79% of the time. Where P(T1) and P(T2) come in is the percentage of time where you receive a Tier 1 report, P(R1). If almost all incidents are Tier 2, you'd expect to almost never see Tier 1 reports (say, 1% of the time). When you do, 79% would actually be Tier 1 events (say, 0.79% of the time are T1 given R1).
P(R1) = P(R1 | T1) * P(T1) + P(R1 | T2) * P(T2)
If you had wanted to do a fancy Bayesian calculation, you could say that the 79% refers not to P(T1 | R1) (the percentage of Tier 1 reports that are actually Tier 1 incidents), but to P(R1 | T1) (the percentage of Tier 1 incidents that are reported as Tier 1). In this case, P(T1) does matter if you want to figure out how to respond.
Given P(R1 | T1), you would need to find P(T1 | R1) (what is the probability that a given Tier 1 report is actually Tier 1?). Using Bayes's rule, you have:
P(T1 | R1) = P(R1 | T1) * P(T1) / P(R1)
We know that P(T1) is 2/17 (almost 0.12) for Tyranitar. Different pokemon would have different probabilities, and that would be important to the result.
P(R1) = P(R1 | T1) * P(T1) + P(R1 | T2) * P(T2) = 0.79 * 0.12 + 0.33 * 0.88 = 0.3841 using the exact values for P(T1) and P(T2).
Remember that here, P(R1 | T2) (the 33%) is the probability that a Tier 2 incident is reported as T1.
So you end up with P(T1 | R1) = P(R1 | T1) * P(T1) / P(R1) = 0.79 * 0.12 / 0.3841 = 0.242. That is, in the case of Tyranitar attacks, if you receive a Tier 1 report, there's just a 24.2% chance that the incident is actually Tier 1.
Does that help?
u/DaystarEld [+2] Pokémon Professor (a day later)
Ok, so I think I get it now... you're basically saying I supplied information that should not be applied to the prior to adjust it. Not without a bunch of extra steps that I didn't do, anyway.
Specifically, the number of T1 events that are reported as T1 is fundamentally different from the number of T1 reports that are actually T1 events, in a way that just taking the ratios and coming up with a "Chance of T1 being reported accurately" from the latter still doesn't translate to "T1 events reported as T1."
So to fix this, I can either add in all the extra math that gives me the actual "Number of T1 events reported as T1," which I can then apply to the prior of 12% Tyranitar T1 events... OR I can just change the information supplied, so say that 79% of T1 events are reported accurately as T1 events, the question becomes:
12% of Tyranitar are Tier 1. 79% of Tier 1 events are reported as Tier 1. 33% of Tier 2 events are reported as Tier 1. A Tyranitar is reported as Tier 1. What's the probability it actually is Tier 1?
Which is then solved like this:
12% of Tyranitar events are Tier 1
88% of Tyraniter events are Tier 2
79% of Tier 1 Events are reported accurately
67% of Tier 2 Events are reported accurately
Group A: 9.48 Tyranitar are Tier 1 and Reported Tier 1
Group B: 2.52 Tyranitar are Tier 1 but Reported as Tier 2
Group C: 58.96 Tyranitar are Tier 2 and Reported as Tier 2
Group D: 29.04 Tyranitar are Tier 2 but reported as Tier 1.
Group A / (Group A+ Group D) = .25% chance a reported Tier 1 Tyranitar is Tier 1
Is that correct? If not I'm happy to get on skype or discord to chat verbally :)
u/masasin [+3] (a day later)
You got it half right. The math is correct (except it's .25 or a 25% chance, not a .25% chance), and this statement is correct:
Specifically, the number of T1 events that are reported as T1 is fundamentally different from the number of T1 reports that are actually T1 events
The ratios you'd done earlier did not actually apply to real life, since they didn't consider the priors. They don't represent the chance of T1 being reported accurately.
The next paragraph is a bit off as well. The alternative (changing the statement) is correct, but the way you can fix that without changing the statement wouldn't be to do any extra math. Instead, it's to just take the percentage of Tier 1 reports which are actually Tier 1 (which you directly provided in the question) as the answer.
Remember, the question was for the percentage of Tier 1 reports which are actually Tier 1. Which you gave. And it does not change by pokemon. It would be the same whether it's 12% Tier 1 or 99% Tier 1.
I'd love to do a Skype call with you. I have Discord but I've never actually tried it for anything. Plus, you're my current favourite serial fiction writer, so it'd be a bonus for me. Do you prefer weekdays or weekends?
u/DaystarEld [+2] Pokémon Professor (a day later)
Oof, okay, I still don't understand this at all then:
Remember, the question was for the percentage of Tier 1 reports which are actually Tier 1. Which you gave. And it does not change by pokemon. It would be the same whether it's 12% Tier 1 or 99% Tier 1.
I don't get why it should be independent of the % of Tyranitar events, whereas in the cancer example it is dependent on the actual cancer rate among the tested population.
I'd love to do a Skype call with you. I have Discord but I've never actually tried it for anything. Plus, you're my current favourite serial fiction writer, so it'd be a bonus for me. Do you prefer weekdays or weekends?
Hey, thanks a lot! Glad you're enjoying the story so much :)
I'm actually heading to CFAR on Monday, and will be there for about a week. I'm available Sunday evening though, if you are, from about 5PM EST till at least midnight.
u/masasin [+2] (a day later)
I don't get why it should be independent of the % of Tyranitar events, whereas in the cancer example it is dependent on the actual cancer rate among the tested population.
You're looking at the wrong thing. When you ask for P(T1 | R1), it's like you're asking what the probability of cancer is given that you had a positive test result. Except, in this case, you gave it outright. 79% of Tier 1 reports are actually Tier 1, in the cancer case would be 7.8% of positive test results are actually cancer patients.
What you didn't give is the marginal probability, P(R1 | T1), which is the probability that a Tier 1 incident is reported as Tier 1. In cancer terms, you didn't give the probability that someone with cancer would get a positive test result, which was 80% in Yudkowsky's example.
Normally, the 80% would hold no matter which percentage of women have cancer. But the way you phrased it, you know that 7.8% of people with positive test results have cancer. If 1% of women have cancer, 80% of cancer patients would get a positive result. If 99% of women have cancer, a much lower percentage would get a positive result, or the false positive would be higher. The effectiveness of the test (with pokemon, the proportion of Tier 1 cases which are reported as Tier 1) will have to change as the priors change in order to keep that 7.8%. That's why, if 7.8% is your given, that number will stay the same no matter which proportion of women have cancer.
To fix that (and here's where the complicated math comes in), you'd need to say that it's 7.8% of women with a positive test result actually having cancer in a population where 1% has cancer. That way, we know the likelihoods, and if that proportion changes to, say, 50:50, we can calculate the changed posteriors.
But here's the thing. You would normally expect the effectiveness of the test, P(+ | cancer) to be the thing that does not change with the population. In pokemon terms, P(R1 | T1), the proportion of Tier 1 incidents reported as Tier 1, would not change with the frequency of severity. What you would expect to change with the prior is P(cancer | +), that 7.8% with 1% cancer ratio. Or, with pokemon, P(T1 | R1), the probability that the report was correct.
tl;dr: You gave the fraction of women with positive mammographies with breast cancer, and asked for it. It's dependent on cancer rates in the population if the test effectiveness is known, but that is not known here. If you provide the posterior as an invariant, the likelihoods would need to change in response to the priors, but the posterior does not change.
I'm actually heading to CFAR on Monday, and will be there for about a week. I'm available Sunday evening though, if you are, from about 5PM EST till at least midnight.
I'll see if I'm free. I'm staying with my family, and they might have events planned. Later is probably more likely. Perhaps after CFAR?
u/DaystarEld [+2] Pokémon Professor (a day later)
Let me try writing out my chain of thought in case there's something simple in it you can point to that would make me recognize how stupid I'm being :)
So for this:
When you ask for P(T1 | R1), it's like you're asking what the probability of cancer is given that you had a positive test result. Except, in this case, you gave it outright. 79% of Tier 1 reports are actually Tier 1, in the cancer case would be 7.8% of positive test results are actually cancer patients.
The question being asked is "How accurate is this test at identifying those with cancer," whereas this:
What you didn't give is the marginal probability, P(R1 | T1), which is the probability that a Tier 1 incident is reported as Tier 1. In cancer terms, you didn't give the probability that someone with cancer would get a positive test result, which was 80% in Yudkowsky's example.
Is asking "How likely is it that someone with cancer will have their cancer properly identified by the test?"
I think I understand that these are two separate things, even if I keep confusing them.
What keeps bothering me is the idea that the Tyranitar ratio is immaterial to how accurate any given test result is, or rather how accurate this particular Tier 1 report is, given that the chance of Tier 1 Tyranitar is very low.
Like, in my head, the fact that T1 Tyranitar are really rare should make the chance that a T1 report is accurate lower because the assumption I have is that people are not well calibrated at determining individual pokemon's threat levels: the 79% accurate Tier 1 reports doesn't mean, in my head, that all events with any given pokemon have the same chance of being accurate. It's an average of ALL reports, where with, say, geodudes, the report accuracy is very high because it's more obvious when it's a T1 vs a T2, but with other pokemon like combee people have a hard time recognizing Tier 2 events, so a lot of their Tier 1 reports are actually Tier 2 events, dragging down the accuracy of general pokemon Tier 1 reports.
So to me, since those false Tier 1 combee reports make up a larger portion of the 21% of Tier 2 events reported as T1, using the 79% accuracy for a T1 combee report would be misleading. A more accurate rate would be the % of Combee Tier 1 reports of actual Tier 1 events, but if not everyone knows that, they just have the 79% to go off of.
And since they're using that more general report statistic, it feels misleading for some pokemon. Some pokemon's individual Tier 1 report accuracy will be closer to that 79% average. Some will be farther. To determine the actual accuracy rate of THIS reported T1 event, it seems like the ratio of Tyranitar T1 vs T2 should actually matter. Like, Tier 1 Tyranitar events are just so rare that this report is inherently less believable, even if most T1 reports are accurate, because most is not all, and so we're a little less confident in this T1 report being accurate than we would be if it's a pokemon with an even amount of T1 and T2 events.
But... as I'm writing this out, now, it feels like I'm recognizing that maybe that's not true, and that what matters isn't how many Tyranitar Tier 1 events there were, like you say, but what the 17 Tyranitar reports were, and how accurate, and then if you have that you can us the ratio of Tyranitar events to determine the actual accuracy of Tyranitar Tier 1 reports, which is the more precise answer to the question of how likely this particular Tyranitar Tier 1 report is to be accurate.
But if you don't have that information, is there really nothing connecting ratio of Tyranitar events to overall accuracy of the Tier 1 reports? If you don't know that Tyranitar reports are less accurate, and all you know is that there were only 2 Tier 1 Tyranitar events in the past 10 years, doesn't that make it an inherently unlikely event that should lower your likelihood to believe its occurrence?
I mean on one hand I get that if there's something super rare but very easy to identify if you know what you're looking for, someone saying they've identified it shouldn't be taken less seriously just because it's rare. But... shit, I mean if someone claims to see a satellite, there's still a higher chance they're wrong than if they claim to see a plane, right? I don't know what % of identified satellites are actual satellites compared to how many satellites get properly identified, but a lot of people don't even know what satellites look like, so their rarity seems intrinsically tied to them being less likely to be properly identified than airplanes, which are seen all the time...
I think I'm rambling at this point and just demonstrating how much I don't get this, since clearly I'm wrong :P But maybe that can help identify where I'm wrong and why. In any case I really appreciate your help, and talking it out after CFAR sounds good.
That said, I'd love to get the chapter fixed before then, so if it's not too much to ask and you have a fairly simple alternative scenario/set of variables for them to demonstrate Bayes' theorem with instead, I'd happily just use that and seek to understand it later.
u/masasin [+2] (a day later)
The question being asked is "How accurate is this test at identifying those with cancer," whereas this:
That's exactly why I avoided using the word "accurate." First, some terminology:
Assume R1 is positive, and R2 is negative. Similarly, T1 is positive, T2 is negative. TP, FP, FN, TN are True Positive, False Positive, False Negative, and True Negative, respectively.
T1 T2 R1 TP FP R2 FN TN
Sensitivity (aka recall or True Positive Rate) = TP / (TP + FN) (# true positive (R1 ∩ T1) / # T1) = P(R1 | T1)
Specificity = TN / (TN + FP) (# true negative (R2 ∩ T2) / # T2) = P(R2 | T2)
Precision (aka Positive Predictive Value, or PPV) = TP / (TP + FP) = P(T1 | R1)
Accuracy = (TN + TP) / (TN + TP + FN + FP) (# correct / # total)
Accuracy is also the sensitivity * prevalence + specificity * (1 - prevalence).
What the first question was asking for is the precision of the test, not the accuracy. Also, note that it uses the actual number of occurrences, and not the probabilities of it happening. That is, you need the priors if you only have the probabilities.
What the second question was asking for is the sensitivity of the test.
Here is Wikipedia with more details.
Anyway, that is why I stuck to P(A | B), which is the probability that A is true assuming B is true. If you can use P(A | B) in your next reply, it'll be much easier to parse and it will cause less confusion. (And, it will make you think of what you're checking for, and what it's relying on, which will help sort things into the right bin.
What keeps bothering me is the idea that the Tyranitar ratio is immaterial to how accurate any given test result is, or rather how accurate this particular Tier 1 report is, given that the chance of Tier 1 Tyranitar is very low.
In the example that's currently in the chapter, you gave the PPV for the Tyranitar incidents, and asked for it back. If you gave the PPV for all incidents, and did not intend it to be constant across all types of incidents, things become much more complicated.
Like, in my head [...] with an even amount of T1 and T2 events.
Aha. This is where things get interesting. What you say about accuracy changing by pokemon does make sense, intuitively. Now, how do you apply that to individual threats?
First things first. Let's assume that the 79% is the average sensitivity, P(R1 | T1), across all pokemon. Different pokemon have different priors. How do the pokemon affect the sensitivity?
- Let's use geodude as something that is perfectly average. It rampages as T1 60% of the time, and T2 40% of the time. P(R1 | T1) in this case is 79%, and P(R1 | T2) is 33%.
- Combee, on the other hand, are cute, so despite their T1:T2 being 30:70, people think it's more innocent than it really is, and almost everyone reports it as R1. P(R1 | T1) is 90%, but P(R1 | T2) is much higher, at 80%.
- Diglett create earthquakes, which are a primal fear. Their T1:T2 is 90:10, but people are scared, so P(R1 | T1) is 30%, and P(R1 | T2) is 10%.
Looking just at this data, you'd expect that, when T1 occurs less often, people report it as T1 more often. Which obviously should not be generalized.
- Tyranitar has a T1:T2 of 12:88, but they're bigger, and scarier, so you might have P(R1 | T1) = 5%, and P(R1 | T2) = 5%.
You wouldn't be able to estimate that kind of stuff just by looking at their T1:T2. In fact, I'm not sure it correlates much, unless the reports on the news raise its danger level in the collective consciousness. That didn't happen with Combee, though, even among trainers, so I'm not sure how valid that idea is. Also, how does it vary by region? Are people in mountainous areas more prone to reporting blizzards as Tier 1 than near the coast? Or are people who were the "cool kids" in class and watched certain shows more prone to recklessness? The best course of action would probably be to use the 79% as default, unless better information exists. Which it would for pokemon with lots of incidents, and then you try and drill down further to get to the location and cultural effects.
Or, if you have extra sources of information, the Rangers can perform a PCA (principal component analysis) and figure out that, I don't know, height and speed is negatively correlated with sensitivity (the taller or faster a pokemon is, the lower P(R1 | T1) becomes), while psychic ability is positively correlated (e.g., if psychic pokemon calm both people and pokemon down, so almost all rampages are T1 and are reported as such). Or, build a neural network and fill it up, which does all that work for you automatically. Then, when you have a new pokemon where the data is lacking, you can use that neural network to figure out the likely sensitivity and specificity for it, and you can use that.
But if you don't have that information, is there really nothing connecting ratio of Tyranitar events to overall accuracy of the Tier 1 reports? If you don't know that Tyranitar reports are less accurate, and all you know is that there were only 2 Tier 1 Tyranitar events in the past 10 years, doesn't that make it an inherently unlikely event that should lower your likelihood to believe its occurrence?
Even assuming a constant 79% sensitivity derived from all incident types, not adjusted for anything, but considering the prior, as well as a 67% specificity, you end up with a 25% likelihood that a Tyranitar T1 actually occurred, so you should (probably correctly) mount a T2 response. In cases like Diglett where almost all incidents are T1, you would find out that both P(T1 | R1) and P(T1 | R2) are very, very high and you'd mount a T1 response regardless of what kind of report you received.
Like, Tier 1 Tyranitar events are just so rare that this report is more likely to be a glitch, even if most T1 reports are accurate, because most is not all, and so we're a little less confident in this T1 report being accurate than we would be if it's a pokemon with an even amount of T1 and T2 events.
That's exactly what's borne out with the calculations even with the 79% sensitivity. It's generally self-correcting, so you don't need to worry about that. If you do have better numbers, then you'd obviously use them, but it will not usually change your response.
Also, don't forget. In a big emergency, you will get multiple reports. After every report, you can use the posterior as a prior and update on that. They probably aren't independent, so it's not a simple Bayesian calculation, but you can probably work out ahead of time how correlated multiple reports from the same area etc are, and factor that in. The point is, if you see 20 R1 and just 2 R2, even for something as big and scary (albeit slow-moving) as Tyranitar that has a high P(T2), you have overwhelming odds in favour of it being that rare T1.
I mean on one hand I get that if there's something super rare but very easy to identify, someone saying they've identified it shouldn't be taken less seriously just because it's rare.
It should be taken less seriously specifically because it's rare. But it shouldn't be dismissed out of hand. You have a default starting probability of 12%. With one report, it got nudged upwards to 25%. Still unlikely, but you can breathe slightly easier. Another R1 comes in.
P(T1 | R1) = P(R1 | T1) * P(T1) / (P(R1 | T1) * P(T1) + P(R1 | T2) * P(T2)) = 44% (I'm just building off from the rounded 25%, not anything exact.)
Suddenly, and R2 comes in.
P(T1 | R2) = P(R2 | T1) * P(T1) / (P(R2 | T1) * P(T1) + P(R2 | T2) * P(T2)) = 20%.
That single R2 at this stage down the probability of it being a T1 almost as much as the two R1 raised it (24% vs 32%). Earlier on, it would have taken P(T1) from 12% to 4%. (The final probability after two R1 and 1 R2 would still end up at 20% - the order of operations doesn't matter.)
Another R1 comes in. You're at 37%. Another. You're at 58% now. You're more cautious. It seems likely that it's T1, but we aren't quite sure yet. (Look up Bayes Factor to see how sure you are. While this not exactly it, in a case like this, you can do 58:42, which is ~1.3, barely worth mentioning.)
Another R1 comes in. You're at 77% now. That's 3.34, and now we have substantial (or positive) evidence.
Two more and you're at 95% confidence. Three, at 98%. You can probably say at either that you have strong evidence that It's a Tier 1 incident.
but if someone claims to see a satellite, there's still a higher chance they're wrong than if they claim to see a plane, right?
It depends. What time were they looking? In which direction? Where were they? Do planes normally fly in the area? If you ask some random person of the population, and they were at night, and there are plenty of planes in the sky, and they don't know that satellites don't tend to blink (I think I saw a tumbling satellite once!), then I'd grant you that there's a high probability that they're wrong. If you let them know that satellites are steady lights moving in a straight line, or showed them a couple of examples before you asked them to report, I doubt they'd get it wrong. And if you'd asked me, I can differentiate between them, can tell you the orbit and altitude, and could possibly tell you the type of satellite or plane it is: I'd be unlikely to be wrong on either.
talking it out after CFAR sounds good.
You could show this thread to the experts there. I'd love feedback too, in case I have it completely wrong.
That said, I'd love to get the chapter fixed before then, so if it's not too much to ask and you have a fairly simple alternative scenario/set of variables for them to demonstrate Bayes' theorem with instead, I'd happily just use that and seek to understand it later.
I think that changing P(T1 | R1) being the given to P(R1 | T1) should be enough, and doing the calculations like you'd done it in the previous post.
u/DaystarEld [+1] Pokémon Professor (2 days later)
I'll probably respond to this more in depth later, but I just wanted to say thanks again for explaining it all out. I think I understand it a little better now, and will be rereading it a few times in the coming days to make sure it sticks a bit.
One thing I wanted to highlight is that this:
Also, don't forget. In a big emergency, you will get multiple reports. After every report, you can use the posterior as a prior and update on that... It should be taken less seriously specifically because it's rare. But it shouldn't be dismissed out of hand. You have a default starting probability of 12%. With one report, it got nudged upwards to 25%. Still unlikely, but you can breathe slightly easier.
Is exactly what I had in mind. The idea in my head is that you start with the prior probability of a Tier 1 Tyranitar (12%) and update from there upward or downward for each report you get based on the general accuracy of reports, or specific accuracy of Tyranitar reports. This is why I was so confused to hear that the prior probability of Tyranitar events has no bearing on P(T1 | R1)... but I guess that's a really bad way for me to phrase it now that I understand what you mean a little better, and clearly the way I represented that mathematically was very off.
Gonna respond to the lower comments with the alternative ways to present the problem now :) Thanks again for the long explanation!
u/masasin [+1] (2 days later)
Yep. You gave us P(T1 | R1), which does not depend on P(T1). If you had P(R1 | T1) instead, you'd need P(T1) to get P(T1 | R1), and you'd use that as your starting probability, P(T1)_2, for the second stage. And then you can repeat as much as you want.
u/daydev [+2] (a day later)
To fix that (and here's where the complicated math comes in), you'd need to say that it's 7.8% of women with a positive test result actually having cancer in a population where 1% has cancer. That way, we know the likelihoods, and if that proportion changes to, say, 50:50, we can calculate the changed posteriors.
But here's the thing. You would normally expect the effectiveness of the test, P(+ | cancer) to be the thing that does not change with the population. In pokemon terms, P(R1 | T1), the proportion of Tier 1 incidents reported as Tier 1, would not change with the frequency of severity. What you would expect to change with the prior is P(cancer | +), that 7.8% with 1% cancer ratio. Or, with pokemon, P(T1 | R1), the probability that the report was correct.
It seems to me what /u/DaystarEld intends is to give us P(cancer | +) for one population (P(T1 | R1) for all Pokemon), then from that calculate effectiveness of the test P(+ | cancer), and then, assuming that effectiveness of the test doesn't change between populations, calculate P(cancer | +) for another population (P(T1 | R1) for Tyranitar specifically). It seems to make sense to me, although in another sub-thread it's argued that his calculation of P(+ | cancer) from P(cancer | +) is wrong.
u/masasin [+2] (a day later)
The way he was calculating it before was wrong, but you could do it the way you're saying. I did it earlier by brute force, though you could probably solve a system of equations for that, since you have the prevalence terms in separate parts of the denominator.
That being said, if you know P(T1 | R1) for a given population, you probably already know P(R1 | T1). And to calculate P(R1 | T1) from P(T1 | R1) in the first place, you'd need P(T1) and P(T2) of that population, which /u/DaystarEld did not give us.
For the narrative's sake, I'd say setting P(T1 | R1) as P(R1 | T1) and asking for P(T1 | R1) as he did is probably the way to go.
I also just made this reply addressing the idea of P(R1 | T1) not changing as P(T1) and P(T2) change.
Anyway, I should be sleeping now. It's 4:18 local time.
u/daydev [+2] (a day later)
And to calculate P(R1 | T1) from P(T1 | R1) in the first place, you'd need P(T1) and P(T2) of that population, which /u/DaystarEld did not give us.
Actually he did, a little bit before giving P(T1 | R1) = 79% they say that P(T1) is 64%.
u/masasin [+2] (2 days later)
Oo, good catch. I'll see if it's possible tomorrow.
u/masasin [+3] (2 days later)
/u/DaystarEld, /u/daydev
Sorry about that! I guess I wasn't paying attention in class. I calculated it for you!
Assumptions:
- P(T1) = 64%
- P(T1 | R1) = 79%
- P(T1 | R2) = 33%
Result:
- P(R1 | T1) = 83.2%
- P(R1 | T2) = 39.4%
Using this data, we can calculate the odds for Tyranitar.
- P(T1) = 2/17; P(T2) = 15/17
- P(R1 | T1) = 83.2%
- P(R1 | T2) = 39.4%
P(T1 | R1) = P(R1 | T1) * P(T1) / (P(R1 | T1) * P(T1) + P(R1 | T2) * P(T2)) P(T1 | R1) = 0.2197, or 22% probability that a Tyranitar attack reported as Tier 1 is actually Tier 1.
You end up doing the same thing, but have that extra step of having to find P(R1 | T1) and P(R1 | T2) manually in the first place.
edit: Please double check for accuracy. It's almost 6 am here with no sleep for me. Basically, write the equations for P(T1 | R1) and P(T1 | R2) and solve for the unknowns. The simultaneous equations if you want to solve them automatically:
You're solving for P(R1 | T1) and P(R1 | T2). Remember that P(T2 | R1) = 1 - P(T1 | R1), and P(T2 | R2) = 1 - P(T1 | R2)
- P(T2 | R1) * P(R1 | T1) + P(T1 | R1) * P(T2) / P(T1) * P(R1 | T2) = 0
- (1 - P(T1 | R2)) * P(T1) * P(R1 | T1) - P(T1 | R2) * P(T2) * P(R1 | T2) = P(T1) - P(T1 | R2) * (P(T1) + P(T2))
u/daydev [+2] (2 days later)
This right here looks like what /u/DaystarEld intended, to make it one step removed from the trivial application of the Bayes rule. Although inside here there's an implicit assumption that P(R1 | T1) & P(R1 | T2) is constant across all Pokemon which seems questionable, as discussed elsewhere some Pokemon may be more or less scary. I think we would need P(R1) & P(R2) for Tyranitar to adjust for that? And since we don't, this calculation is the best we can do with the available data.
UPD: I'm also not too sure how conducive it is for the educational purpose to complicate the introduction of the Bayes Rule with extra steps. But on the other hand it would be quite unbelievable if Red failed to just plug numbers into the Bayes formula.
u/masasin [+1] (2 days later)
Red is bad at math. If /u/DaystarEld could plug the wrong numbers in, Red could too.
I'm also not too sure how conducive it is for the educational purpose to complicate the introduction of the Bayes Rule with extra steps.
I agree with this. I stand by my recommendation that P(R1 | T1) and P(R1 | T2) be given.
u/DaystarEld [+1] Pokémon Professor (2 days later)
Thanks for all your help!
u/Iijil [+2] (2 days later)
I think the first equation should have a - instead of the +. In the second we can use P(T1)+P(T2) = 1.
Corrected:
- P(T2 | R1) * P(T1) * P(R1 | T1) - P(T1 | R1) * P(T2) * P(R1 | T2) = 0
- (1 - P(T1 | R2)) * P(T1) * P(R1 | T1) - P(T1 | R2) * P(T2) * P(R1 | T2) = P(T1) - P(T1 | R2)
Although I think it is easier to calculate P(R1) explicitly in an additional step. By doing that we can solve three equations, one after the other instead of solving a system of two equations simultaneously.
We get P(R1) from P(T1) = P(T1 | R1) * P(R1) + P(T1 | R2) * (1 - P(R1)), where it is the only unknown. After we have that we use P(R1 | T1) = P(T1 | R1) * P(R1) / P(T1) and P(R1 | T2) = P(T2 | R1) * P(R1) / P(T2)
For the purposes of the story it is probably clearer and more instructional if P(R1 | T1) and P(R1 | T2) are known from the lecture.
edit: For the actual numbers, I get P(R1 | T2) = 39.3% Probably a rounding difference of some kind, but rounding more accurately I get .39311594...
The rest of the numbers stays the same (excluding differences that are rounded away).
u/masasin [+1] (2 days later)
I got 39.3% and 39.4%. Instead of rerunning it with a smaller step, I chose 39.4% because it seemed closer. You're probably more accurate. Thanks for the simplifications and correction.
For the purposes of the story it is probably clearer and more instructional if P(R1 | T1) and P(R1 | T2) are known from the lecture.
Agreed.
u/DaystarEld [+1] Pokémon Professor (2 days later)
We get P(R1) from P(T1) = P(T1 | R1) * P(R1) + P(T1 | R2) * (1 - P(R1)), where it is the only unknown.
Assuming the P(T1) you're referring to is the overall amount and not the Tyranitar amount, is P(R1) = ~21%? Or did I screw that math up?
u/Iijil [+1] (2 days later)
Yes, P(T1) there is for the overall amount. But 21% is not right. I get ~67%.
Plugging in the numbers:
.64 = .79 * P(R1) + .33 * (1 - P(R1))
.64 - .33 = (.79 - .33) * P(R1)
P(R1) = 31/46
u/DaystarEld [+1] Pokémon Professor (2 days later)
Thank you for all your help!
u/DaystarEld [+1] Pokémon Professor (2 days later)
To clarify something real quick, how is
P(R1 | T1) = 83.2%
Found again? I thought it was the proportion of correct T1 reports over all T1 Reports, but get the wrong answer when I try it.
u/masasin [+2] (2 days later)
Look at this reply by /u/Iijil. He fixed my derivation, and had the brilliant idea of finding P(R1) first.
Let's work through it step by step:
P(T1) = P(T1 | R1) * P(R1) + P(T1 | R2) * (1 - P(R1))
Solve for P(R1)
P(R1) = (P(T1) - P(T1 | R2)) / (P(T1 | R1) - P(T1 | R2)) = (0.64 - 0.33) / (0.79 - 0.33) = 0.6739
Now, we solve for P(R1 | T1) and P(R1 | T2)
P(R1 | T1) = P(T1 | R1) * P(R1) / P(T1) = 0.79 * 0.6739 / 0.64 = 0.8319
P(R1 | T2) = P(T2 | R1) * P(R1) / P(T2) = 0.21 * 0.6739 / 0.36 = 0.3931
Now, we solve for Tyranitar (first report).
P(T1 | R1) = P(R1 | T1) * P(T1) / (P(R1 | T1) * P(T1) + P(R1 | T2) * P(T2)) = 0.8319 * (2/17) / (0.8319 * (2/17) + 0.3931 * (15/17)) = 0.22
edit:
What we just did here, though, doesn't have much to do with Bayesian reasoning apart from the fact that it uses Bayes's rule. It's just algebra for the most part.
For the story, I recommend that you use the 79% as P(R1 | T1) instead of P(T1 | R1), and treat it as given and constant. If you want to make it such that they do perform updates, you can give them a sequence of R1 and R2 and have them adjust their probabilities accordingly, which is what we did in the last part of this comment. And this is, as you'd said earlier, the essence that you wanted to get across. A few bits of evidence don't do much on their own. Sure, you update your beliefs (up to 25% instead of 12%), but you don't change your behaviour. You still respond as if it's a T2 (if it's binary) or interpolate. The more evidence you get, though, the higher your belief (98%, in the last case) that it is just a T1 incident, and respond appropriately.
u/masasin [+1] (2 days later)
/u/DaystarEld See the edit above.
u/DaystarEld [+1] Pokémon Professor (2 days later)
Thanks for the detailed runthrough :)
And this is, as you'd said earlier, the essence that you wanted to get across. A few bits of evidence don't do much on their own. Sure, you update your beliefs (up to 25% instead of 12%), but you don't change your behaviour.
Right, but I can have them say something like "So it's still not a high chance, and we need to wait for more T1 reports to update further" or similar.
u/masasin [+1] (2 days later)
That sounds good. Note that you need to wait for more reports (in general) to update further, because you can update up (R1) or down (R2).
u/DaystarEld [+1] Pokémon Professor (2 days later)
Right, though generally Rangers have a limited amount of time to respond to a report, so they'll only really get a handful at most before they've got to act on something.
u/masasin [+1] (2 days later)
That's fine. I wonder if interpolation is feasible, in that case? e.g., we're 30% confident that it's T1, so we send everyone who's supposed to respond to Tier 1 incidents, and 70% (100% - 30%) of the reserves designated to deal with Tier 2 incidents.
If there are 5 people normally assigned for a T1, and 30 people for a T2, you'd end up sending 5 + (30 - 5) * 0.7 = 22.5, or 23 people. You still have seven people left who can deal with another T1.
Or you could scale it differently. If you use the log of the difference, it might go up quickly the lower it is, so that by the time you get to 50% confident about T2, you have 95% manpower. Something like this image, where the blue line represents the original interpolation, and the green line the log interpolation.
u/DaystarEld [+1] Pokémon Professor (2 days later)
Something like that sounds like a good idea for a way to improve the Ranger system as it currently is. Though the interpolation scale will probably be different, yeah, since there's not much added benefit in ramping up quickly then tapering off as you approach a higher chance of it being T2: instead they'd want to keep the amount they commit as small as possible for low or even moderate chances of T2 and then ramp up very quickly in the, say, 70-90% confidence range.
u/masasin [+1] (2 days later)
So something like an exponential (e.g., 2^x etc) or polynomial (e.g., quadratic (x^2 etc)) function then.
u/DaystarEld [+1] Pokémon Professor (2 days later)
Just updated the chapter :) If there's anything else wrong with it now I'll have to wait to fix it later: I'm off to bed so I can get to the airport on time tomorrow. Thanks again for all your time and effort: I'll message you when I'm back in town so we can schedule the chat!
u/masasin [+1] (2 days later)
I'd like that. RemindMe! 1 week.
u/RemindMeBot [+1] (2 days later)
I will be messaging you on [2017-12-11 06:05:57 UTC](http://www.wolframalpha.com/input/?i=2017-12-11 06:05:57 UTC To Local Time) to remind you of this link.
[CLICK THIS LINK](http://np.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=[https://www.reddit.com/r/rational/comments/7gujj6/rst_pokemon_the_origin_of_species_ch_50_comfort/]%0A%0ARemindMe! 1 week.) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](http://np.reddit.com/message/compose/?to=RemindMeBot&subject=Delete Comment&message=Delete! dqqli9q)
^(FAQs) [^(Custom)](http://np.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=[LINK INSIDE SQUARE BRACKETS else default to FAQs]%0A%0ANOTE: Don't forget to add the time options after the command.%0A%0ARemindMe!) [^(Your Reminders)](http://np.reddit.com/message/compose/?to=RemindMeBot&subject=List Of Reminders&message=MyReminders!) ^(Feedback) ^(Code) ^(Browser Extensions)
u/DaystarEld [+1] Pokémon Professor (11 days later)
I'm back in town, feel free to add me on Skype as Daystar Eld and we can get around to scheduling the chat :)
u/masasin [+1] (11 days later)
Sure. I'll do that soon.
u/masasin [+1] (2 days later)
I didn't do the calculations again, but this looks good. I love how you used your original error as Red's, and you fixed it well, too. I also like that you used two completely different methods to get the same answer.
Plus, you incorporated Bayesian updates, and potentially opened a way for Leaf to figure out a new (world-changing?) method of deploying Rangers.
u/daydev [+1] (2 days later)
I had to find the actual probability of Tier 1 events first. Look, here...
Should probably say "probability of Tier 1 reports".
See how that's different than "73% of Tier 1 reports are Tier 1 events?" she asks.
Should probably say "83% of Tier 1 reports" since that's the probability she calculated.
Other than that looks good.
u/DaystarEld [+1] Pokémon Professor (3 days later)
Fixed the first one, thanks! For the second one, she's referring to his mistake of treating (R1 | T1) as (T1 | R1), so I actually should have written 79% :) Fixed that too.
u/Iijil [+2] (11 hours later)
By considering 100 reports for T1 and 100 reports for T2 you are implicitly assuming that both reports are equally likely. Which can't be true given the actual event classifications and false report ratios provided. To properly combine them you'd need to adjust the report numbers to match the actual ratio between reports for T1 and T2.
67 T1 reports to 33 T2 reports seems to fit the data from the lecture okay. I found those by solving the system of linear equations given by the false report rates and event classification rates. Out of 100 reports you get 53 T1s reported as T1, 11 T1 reported as T2, 14 T2 reported as T1 and 22 T2 reported as T2
u/DaystarEld [+2] Pokémon Professor (11 hours later)
By considering 100 reports for T1 and 100 reports for T2 you are implicitly assuming that both reports are equally likely.
I'm still not sure why this matters: isn't this like saying "You need to know how many times the cancer test is administered before you can make use of the information that it's 99% accurate?"
(Outside of the general epistemic value in knowing sample sizes that justify accuracy rates, I mean)
u/Iijil [+2] (12 hours later)
Well, it matters because choosing different ratios of reports to consider you get different result.
200 Events reported as Tier 1
158 are actually Tier 1
42 are actually Tier 2
100 Events reported as Tier 2
67 are Tier 2
33 are Tier 1
would combine with the logic from the chapter to
Chance of Tier 1 being reported accurately = 158 / (158 + 33) = .83
If there is a difference depending on the ratio the ratio matters. To figure out which we should use we can work out the full report/actual square that fits the given probabilities. As it turns out using the actual ratio between reports gives the correct result.
In the cancer analogy this is like only knowing the prior probability of having cancer and the probability of having cancer given the different test results and trying to work out the accuracy of the test from that. To do so we do need to consider the ratio of positive/negative answers in some manner.
u/DaystarEld [+2] Pokémon Professor (18 hours later)
Ok, I see how a different amount of Tier 1 reports changes the outcome. Can you walk me through what makes this situation different from the cancer one?
Instead of this:
1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?
I have this:
12% of reported Tyranitar events are Tier 1. 71% of Tier 1 events will be reported as Tier 1. 24% of Tier 2 events will also be reported as Tier 1. A Tyranitar has been reported as Tier 1. What is the probability that it is actually Tier 1?
Neither mentions the amount of actual women who get tested, so why does it matter for the second one? Is it because in the story example there's a completely separate test (Tier 2 reports) that can give a false positive, whereas in the cancer example the same test will either give the false negatives or the false positives?
Edit:
Or is it because of the age group thing? Am I essentially saying:
1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman not in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?
By mixing non-Tyranitar-specific "Tests" while the prior is based specifically on Tyranitar?
u/daydev [+3] (21 hours later)
I have this:
12% of reported Tyranitar events are Tier 1. 71% of Tier 1 events will be reported as Tier 1. 24% of Tier 2 events will also be reported as Tier 1. A Tyranitar has been reported as Tier 1. What is the probability that it is actually Tier 1?As I understand, if we have this, it's no different. What is contested is how we get these probabilities from the other reverse ones. The medical test analogy for what we get in the story would be if it was formulated something like this:
1% of women at age forty who participate in routine screening have breast cancer. 80% of women who get positive mammographies actually have breast cancer. 9.6% of women who get negative mammographies also have breast cancer. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?
Using the methodology in the story, we would divide 80 / (80 + 9.6) and get 89.3% to plug into the Bayes formula. As I understand, it's contested, and it seems intuitively right, that this methodology is wrong, and we can't just divide 80 / (80 + 9.6), we need to consider how many positive and negative results there was in total.
u/Iijil [+3] (a day later)
u/daydev has it essentially right.
First we have a flip in the direction of the conditions. To change that around we need the ratio of reports.
Second we apply the test to some different group, where we have some reason to believe the accuracy of the test will stay the same. The difference between that group and the original group is in the prior.
Last we use Bayes to flip the conditional direction again, because that is the actually useful direction to use.
The best analogy for the cancer situation I can construct is this:
1% of women at age forty who participate in routine screening have breast cancer. 2% of women at age sixty who participate in routine screening have breast cancer.
8% of women at age forty who get positive mammographies turn out to actually have breastcancer. 0.2% of women at age forty who test negative turn out to still have breast cancer.
We think the test will have the same accuracy for both age groups.
A woman at age sixty had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?
To answer that question we first find the accuracy of the test by only looking at the age 40 group. There we have: P(c)=1%, P(c|+)=8%, P(c|-)=0.2%. We are looking for P(+|c) and P(+|no c).
I'm not sure if there are better ways to go about this, but the best approach I know is to first figure out P(+). We know that P(c)=P(c|+)P(+)+P(c|-)P(-) and P(-)=1-P(+). We solve for P(+)=(P(c)-P(c|-))/(P(c|+)-P(c|-)) and get about 10.256%. Once we have that we can use it to flip the direction of the conditions by using it as the prior in a Bayes calculation. We get P(+|c)=82% and p(+|no c)=9.5%.
This is the point in the calculation where the example in your edit starts out. We know the accuracy of the test, the prior probability of the age group we want to apply it to and now want to figure out the actual chance of having cancer once we test positive.
Since we assume the accuracy between age groups stays the same we can now just use the calculated accuracy with the known prior chance of having cancer for the age sixty group to apply Bayes again. We have P(c)=2%, P(+|c)=82% and P(+|no c)=9.5% so we get P(c|+)=15%.
Soooo, the difference, where we need to know the ratio between reports of Tier 1 and Tier 2 is in the step both of your examples skip over, where we turn the initial conditional direction around. The ratio between incoming reports is important, not their actual number.
There may or may not be clever mathematical ways to get the result without calculating the ratio in between, but I don't know of them.
There is no mathematical reason that keeping the accuracy of the reports the same between general Pokémon and Tyranitar is the correct thing to do. That part is taken from additional reasoning about the world.
u/DaystarEld [+1] Pokémon Professor (2 days later)
There is no mathematical reason that keeping the accuracy of the reports the same between general Pokémon and Tyranitar is the correct thing to do. That part is taken from additional reasoning about the world.
Right, by my reasoning they wouldn't be the same, but if all you have is the accuracy of general pokemon reports and the ratio of Tyranitar events, I thought both could be used to reach an estimate of how likely a Tyranitar report is to be accurate. Apparently I was super wrong XD
u/daydev [+2] (20 hours later)
79% of Tier 1 events will be reported as Tier 1. 33% of Tier 2 events will also be reported as Tier 1.
But the text presents it differently, doesn't it?
Reports Tier 1 are actually Tier 1 79% of the time. Reports of Tier 2 are also actually Tier 1 33% of the time.
That's like saying:
Out of women who get positive results 80% actually have cancer. Out of women who get negative results, 9.6% also actually have cancer.
The difference in the direction of causality between P(R1 | T1) & P(T1 | R1).
u/DaystarEld [+2] Pokémon Professor (20 hours later)
Woops, fixed.
u/masasin [+2] (11 hours later)
I just did a brute force search. I tested it first using the default values for Yudkowsky's original medical test.
Yudkowsky's test:
- P(Cancer) = 0.01; P(Healthy) = 0.99
- P(Cancer | +) = 0.0776; P(Cancer | -) = 0.0022
If you brute force for P(+ | Cancer) and P(+ | Healthy), stepping at 0.001 (0.1%) and considering it a match if there's a difference of less than 0.01%, you end up with the following:
P(+ given Cancer) P(+ given Healthy) 0.799 0.096 0.800 0.096 0.807 0.097 0.808 0.097 0.809 0.097 The "real" answer was 0.8 and 0.096, respectively (the second row).
Trying to solve your problem, I found no matches even if I increased the tolerance to a whole percent. If we give a tolerance of 10%, you do find solutions when:
- P(R1 | T1) >= 0.876
- P(R1 | T2) >= 0.945
This gives you P(T1 | R1) of 0.11 to 0.12, and P(T1 | R2) of anywhere between 0.2-ish to 0.5-ish.
In other words, if 21% of Tier 1 reports are wrong, and 33% of Tier 2 reports are wrong, you need:
- 87.5%+ of Tier 1 threats are accurately reported as Tier 1 (definitely plausible), AND
- 94.5% + of Tier 2 threats are incorrectly reported as Tier 1 (very strange)
However, if you do assume that these conditions hold, then a Tyranitar rampage reported as Tier 1 would actually be Tier 1 around, say, 12% of the time.
If you know what values you'd like for P(R1 | T1) and P(R2 | T2) for all kinds of attacks (e.g., 60% of Tier 1 threats are reported as Tier 1, but 80% of Tier 2 threats are reported as Tier 2), you can calculate the posterior probability of a report having gotten it right using forward Bayes.
For example, assuming a constant 60/80, no matter the kind of threat:
- If 50% of threats are Tier 1 and 50% Tier 2, a Tier 1 report would be accurate 75% of the time, and a Tier 2 report would be accurate 67% of the time.
- If, like Tyranitar, 11.8% are Tier 1 and 88.2% are Tier 2, a Tier 1 report is actually Tier 1 only 28.57% of the time, and a Tier 2 would be accurate 93.75% of the time.
- If, like Digglett, say 90% of incidents are Tier 1, a Tier 1 report is correct 96.4% of the time, while a Tier 2 report is accurate only 18.18% of the time.
edit:
If you want to use 0.79 and 0.67 as the likelihoods (instead of 79% of Tier 1 reports actually being Tier 1, you have 79% of Tier 1 incidents being reported as Tier 1), with Tyranitar, you end up with:
- P(T1 | R1) = 0.242
- P(T2 | R1) = 0.758
- P(T1 | R2) = 0.040
- P(T2 | R2) = 0.960
In other words, a report of Tier 1 knowing that it is Tyranitar would just have a 24.2% chance of actually being Tier 1. Therefore, you should consider it a Tier 2, or mount a tiered response (if you normally send 5 rangers to Tier 1, and 30 rangers to Tier 2, you should send 5 * 0.242 + 30 * 0.758 = 24 rangers if you have a Tier 1 report, and 5 * 0.04 + 30 * 0.96 = 29 rangers if you have a Tier 2 report).
u/Ristridin1 [+2] (2 days later)
Just adding another reason why the math in the current chapter doesn't work (didn't quite know where to put it in the ongoing thread):
100 Events reported as Tier 1
79 are actually Tier 1
21 are actually Tier 2
100 Events reported as Tier 2
67 are actually Tier 2
33 are actually Tier 1
If you used the above data, out of 200 reports, 79+33 = 112 would be Tier 1, while 88 would be Tier 2. This clearly doesn't match the actual 2:15 ratio of Tier 1: Tier 2.
Here's another flaw that I just noticed (apologies if it was already pointed out). You are using the 79% and 67% incorrectly in the above argument. The 79% accuracy says that out of 100 actual Tier 1 events, 79 will be reported as Tier 1 (more technically, P(Tier 1 reported given there is an actual Tier 1 event) = 0.79). On the other hand, in the above data, the claim is that out of 100 reported Tier 1 events, 79 were actually Tier 1 (or P(actual Tier 1 given that Tier 1 is reported) = 0.79). These are not the same things.
On an aside, this gives a way to switch the chapter around (and possibly preserve Red's flaw). Instead of mentioning actual Tier 1 and Tier 2 events, mention there were 6 Tier 1 reports and 11 Tier 2 reports, and then try to figure out how many were actual Tier 1. Using flawed reasoning (flipping the conditional probabilities), Red can imagine that 60.79 reports were actual Tier 1 reported correctly, while 110.33 were actual Tier 2 reported incorrectly. This gives 4.74 + 3.63 = 8.37 of Tier 1, or 8.37 out of 17 events as Tier 1 (about 49.2%).
In actuality, there were X actual Tier 1 events and 17-X actual Tier 2 events. This would give 0.79*X + (17-X)*0.33 reported Tier 1 events, or simplified: 5.61 + 0.46X reported Tier 1 events. Since there were 6 reported Tier 1 events, X should be 39/46, or about 1 actual Tier 1 event.
u/DaystarEld [+2] Pokémon Professor (2 days later)
If you used the above data, out of 200 reports, 79+33 = 112 would be Tier 1, while 88 would be Tier 2. This clearly doesn't match the actual 2:15 ratio of Tier 1: Tier 2.
That's because the 2:15 ratio is for Tyranitar, though, while the hypothetical 200 reports used to generalize the accuracy %s is for all pokemon events reported.
These are not the same things.
Right, I changed it for this example to keep the numbers the same out of convenience just to see if I understood the issue.
Instead of mentioning actual Tier 1 and Tier 2 events, mention there were 6 Tier 1 reports and 11 Tier 2 reports, and then try to figure out how many were actual Tier 1.
This is a good idea, but for now what I want to ideally preserve the ultimate question presented: upon hearing that a report is T1, how do you figure out the chance if it's actually tier 1 or not? If I have to change the other bits of information provided for that, I'm happy to :)
u/Ristridin1 [+1] (9 days later)
I just saw the new version, and there's still something odd. Namely, as far as I can see, the total probability of a Tyranitar Tier 1 event can not possibly be 2/17 with the given conditional probabilities.
I'll write T1 for Tier 1 event, TT1 for Tyranitar Tier 1 event, R1 for Tier 1 report, TR1 for a Tyranitar event reported as Tier 1 (and likewise T2, TT2, R2, TR2).
Here's the argument: You know P(TT1) = P(TT1 | TR1)*P(TR1) + P(TT1 | TR2)*P(TR2). Assume P(TT1 | TR1) = P(T1 | R1) = 0.79 and P(TT1 | TR2) = P(T1 | R2) = 0.33 (this is where things will go wrong). We have P(TR1) = 1-P(TR2). We find P(TT1) = 0.79 X + 0.33 (1-X) = 0.33 + 0.46X. Since X is between 0 and 1, at least 33% of all Tyranitar events should be Tier 1, which is larger than 2/17. Actually, I didn't need to do this computation at all: If 33% of reported Tier 2 events are Tier 1 and 79% of reported Tier 1 events are Tier 1, then at least 33% and at most 79% of all events are Tier 1.
The reason for different notation for Tier 1 and Tyranitar Tier 1 should now be clear. In the chapter, Leaf starts with P(T1) = 0.64 and then moves on to P(T1) = 2/17. The latter should be P(TT1) = 2/17.
Of course, there's a reasonable argument to be made that P(TT1|TR1) does not equal P(T1 | R1) = 0.79 etc. But in that case, your results from the calculation of P(R1|T1) etc. can not be used to say anything about P(TR1 | TT1).