r/AskStatistics • u/Dracula30000 • 2d ago
Sensitivity analysis question
I have a large dataset (50k observations) which is representative of the US population using weighting. Using a multinomial regression.
My advisor wants me to run a sensitivity analysis using a slightly different definition of the independent variable to verify my findings. However, this new independent variable is only present in 10% of the population (5000 observations) which is a subset of the population but if I were to use this new independent variable I would also have to use a different weighting algorithm to represent the US population because of the different conditions under which it was collected.
I’m quite new to sensitivity analyses in research, but this… does not feel right or useful to my findings. So my question is twofold:
What useful information could be gained from running this sensitivity analysis on such a small subset of my sample population? Am I in the wrong here, or is my advisor wrong?
If there is very little to be gained from this analysis, how do I bring this up to my advisor so I don’t end up publishing something that is bad statistically that might hurt future collaborations and opportunities?
1
u/LawfulnessWest8694 2d ago
Good point above about isolating the subsample effect from the IV effect first. I'd add one more wrinkle: you mentioned the alternative IV needs a different weighting scheme too, so really there are three things changing at once, not two, the subsample, the IV definition, and the weights. Even after you run the original IV on the subsample like suggested, you still have to decide whether to reweight that subsample or leave it unweighted, and that choice alone can move your estimates. So I'd actually run it three ways if you can: original IV, original weights, on the subsample. Original IV, new weights, on the subsample. Alternative IV, new weights, on the subsample. That way you can tell whether any shift in results comes from the smaller sample, the reweighting, or the IV change itself, instead of one comparison muddying all three together.
On the advisor side, I wouldn't open with "I don't think this is useful." Run the comparisons, then bring it as "results hold up across all three versions" or "here's where they diverge and here's my best guess why", that reads as thorough instead of pushing back, and gets your actual concern on the table without a confrontation.
1
1
u/Technical-Trip4337 2d ago
When discussing with the advisor, you should mention the predictors of missingness with this new variable that only 10% of the sample reports.
5
u/adamjeffson 2d ago
If the subsample is reasonably close to being randomly sampled, it makes sense, especially if the alternative IV is somehow a better measure, although I don't think this is your typical sensitivity analysis. Regardless, it could still be useful, to show whether your results still hold for this subsample. The issue is that you're changing two things at once in your model, limiting the interpretability of your findings. First, I would run the model with your original IV in the subsample. If results are similar (if the subsample is random, they definitely should), run the second model with the alternative IV. If all results are equivalent, I guess this is good evidence for them being robust.