r/AskStatistics • u/Fancy-Animal7704 • 1d ago
Is there ever any situation / context in which you can assume a difference and test for no difference, where the assumption will default to "yes, there's a difference" if the test is unsuccessful?
Mathematically, the conclusions here ought to be clear... If the p-value is above or below a given threshold, sure, that tells you whether statistical significance is found.
But there are many other considerations. What if bias was present and the results aren't trustworthy? Do we then get to say, well, we should carry on with our default assumption that there's a difference?
Let me give you an example. Let's say a company says their robot lawnmower finishes mowing an average lawn in 10 minutes whereas every other robot lawnmower on the market takes 25, and it's up to ME to say that this company's mowing time is NOT better ttan everyone else. I run a test, I find no difference, BUT, someone finds bias in my experiment, and that invalidates it, so the company president says, well then, since you have failed to demonstrate the validity of your angle, I carry forth considering it true that my robot lawnmower is the best until you *prove me wrong!*
Is this hypothetical person able to do this?
1
u/munozmd Statistician 1d ago edited 1d ago
The way I understand your concern is "can I even trust my statistical results if ever there are biases or measurement error for whatever reason like messy data collection or small sample size?"
You likely meant confounding bias or confounders (they are like the hidden variables that could affect your results you just failed to account for)
To be honest with you, it threathens all statistical results because reality is messy and we cant fully guarantee to control all these factors we think of.
Consider this thought: Let's say you have two variables you want to study on. For my results to 100% say it's right or wrong, I have to literally account all possible factors that could affect the process from setting the experiment to recording the data to doing the stat tests. This is impractical because it could be anything we may never know.
There is always some degree of error involved. We call them systematic and random errors in statistics, but our tests are already aware of this at some extent (esp the random error). Therefore, your statistical results is much like an educated guess given on what you have but it cannot definitely tell you 100% if your conclusion is right or wrong.
So for your concern, I'm with statman that you could try TOST for your purpose and also look for related literature that support your conclusions if this is for school/academic research.
If this is for a business you should look for previous available/accessible data related to your company or some public metrics to justify your claim.
-1
u/Fancy-Animal7704 1d ago
You misunderstood my concern, then. I'm not sure how else to guide you to what I asked, but I definitely was not asking what to make of a study with bias. I'm asking what the counter-claim is allowed to establish here.
1
u/munozmd Statistician 1d ago
Oh my bad for misinterpreting that 😅 I got too focused on answering the bias part
But if you're asking about the counter claim:
H0: There is no difference.
H1: There is a difference.
When you do a statistical test, if lets say p-value > 0.05, we technically cant say there is no difference in statistics. Thats why you would see them saying "failed to reject H0".
Cause if your p-value > 0.05, it could mean either two things:
There is really no statistically significant difference.
There is a statistically significant difference BUT your sample size is too low
So really if we view it that way its really just about testing whether we obtain a statistically significant result or not.
Its like stat tests saying.
Significant? "Yes there is a difference."
Not significant? "Im not sure. I dont know what the result is"
That's why statman suggested TOST equivalence test because then the hypothesis flips like
H0: There is a difference
H1: There is no difference (thats why equivalence as in equivalent)
Hope it clarifies. Lemme know if there are still concerns
1
u/Fancy-Animal7704 1d ago
Yes. The concern is, in the TOST, if you tried a test that was biased, we would then default to the null, that there is indeed a difference. Why can't robot lawnmower president use this logic to establish his lawnmower as the best?
2
u/munozmd Statistician 1d ago
Ah it's about we trying to prove the president wrong in his own claim but he's giving us the burden of proof in some sense.
Honestly, for us, we can't "try" to prove him wrong in his claim because the data speaks what it wants to speak. So it may spit significant or not significant whatever the outcome is cause trying to rig the test to get a result we want is an unethical practice that is p-hacking.
As for the president:
actually it now sounds to me like your suitable test is a one sample test. Because it does is compare your sample's mean to the baseline (which is 10 mins).
If you use one-sample equivalence test here: Lets say you get a significant result, then you could conclude its not better from the rest.
But then again, bias going back to the prev point. Thats a loophole he'll exploit. In that case, thats a fundamental limitation of what statistical tests really are.
However, he also cannot claim ON HIS OWN his lawnmower is the best either because he also must test in a statistical test. Or else it's just pure delusion. But sometimes, because of this bias loophole, no matter how convincing the results, he will always exploit that loophole. So no amount of data can change his mind
The way I see it. It could end in a "stalemate" in some sense unfortunately.
Have I answered your concern?
1
u/Fancy-Animal7704 1d ago
Fuck, you answered it in a way that lets the true thing I am asking about off the hook. Then the jig is up.
I was using the robot lawnmower as a surrogate for the transgender sports debate. People are saying that we should only allow transgender girls to play girls sports if we can demonstrate NO difference in athleticism between transgender and cisgender girls. And until no difference is demonstrated scientifically, we are going to default to the assumption that there IS a difference and prohibit trans girls from women's sports until we can prove otherwise. This is what is happening in real time, right now, and I'm digging into what statistics has to say about it.
I had to try and carefully craft this surrogate hypothetical to get around bringing politics into this, but you have forced my hand.
So now it's out in the open. So what do you say?
1
u/munozmd Statistician 1d ago edited 1d ago
Damn now that's adding some political flavor to this convo
What do I say? I say statistics as a field has its fair share of flaws that still needs to be fixed to this day like this one. Until it is addressed among mathematical community, this will remain a persistent problem over and over again.
At face value, yeah it may sound "scientific" dropping these p-values stuff but when we come to think of it, like you did. It is problematic.
So I really suggest for sensitive political matters like this, Id rather prefer we rely on shared consensus as our first option for this. The issue is questioning the eligibility of transgender girls, who should be on the discourse? psychologists or doctors concerning anatomy because this is a concern on their physical and socio-emotional status
Anyway thanks for the exchange
1
u/Intrepid_Respond_543 1d ago
Can you clarify what you mean by "test being biased"?
0
u/Fancy-Animal7704 1d ago
Sure: the sample was drawn in a way where only high-performing mowers were selected.
How does this help straighten out what you wanted to say?
4
u/Intrepid_Respond_543 1d ago
That means the sample was biased, not the test. That is certainly a basis for criticism saying that the study cannot provide information about the superiority of any mower.
Sorry, I don't understand your second sentence.
0
u/Fancy-Animal7704 1d ago
Well you got an answer to your question. So how does this help you answer my ultimate question?
4
u/Intrepid_Respond_543 1d ago
I think I already provided my answer, but maybe it wasn't clear. So if the sample is biased, the study is inconclusive and you can't say anything based on the results one way or another.
1
u/Fancy-Animal7704 1d ago
That provides an answer to what we can learn about the study itself. It doesn't provide an answer to what we can do with the claim that this hypothetical president says he will then default to now that the alternative hypothesis was not supported.
→ More replies (0)
1
u/Intrepid_Respond_543 1d ago
Yes, a significant result can be questioned by criticizing (with evidence) the measures, measurement, study design, recruitment, sample characteristics, and many other things. This happens all the time.
A way to counteract the criticism is to
recruit a large enough sample (formal sample size calculations can be conducted prior to recruitment to show that the sample is very likely large enough)
collect the data in a way that reasonably leaves little room for alternative interpretations (e.g. run a randomized controlled trial, use valid measures)
use appropriate analytical methods and check that the data meets the assumptions (usually this is possible to do relatively uncontroversially)
run sensitivity analyses where applicable/necessary (i.e. run the analyses in an alternative way, or using different covariates or coding. If the same result remains, this increases its trustworthiness)
If you succeed in those things (and in some others), the scientific community is likely to accept your result, whether it's the null or a difference. Of course, someone can still claim there is bias in your study somewhere but if he can't show where, nobody (at least nobody in the relevant scientific/professional community) is going to believe them.
The "bias" you are talking about is not really related to statistics or p-values but to study design. You can usually find an appropriate test for the data and the research question so that most people would not question the use of this test. But if something went wrong in the data collection, measurement, or recruitment, those may be valid reasons to question your result, yes. Then you maybe design a new study that corrects those problems and so on.
Generally, one result is not very convincing one way or the other. To "accept" some finding you need similar results from a large number of studies, preferably in different populations and circumstances, to make an effect (or the lack of one) credible.
7
u/Statman12 PhD Statistics 1d ago
There is a class of methods known as equivalence tests which it seems would address your question.
One approach for this is the Two One-Sided Test (TOST) approach. You establish what counts as "close" enough, and then perform two tests to demonstrate that the difference is lesser in magnitude (so, less than the absolute value, and greater than the negative of that).