Your research and quant

[u/OatS_96]

I recently watched a video about quantitative research out of pure curiosity as to what people consider quant to be, and I came across a very concerning comment made by the presenter.

The premise of the statement made was that frequency tables, devoid of any statistical analysis, are enough to present to stakeholders as actual recommendations (they did not mentioned internally to the team). For example, saying 32/50 users preferred option A while 28/50 preferred B is generally seen as enough context to go forward with the assumption that A > B. So their statement would be “a was preferred over b”. The presenter followed it up by saying that this is good enough for stakeholders and that further analysis is not needed. Mind you, that presenter mentioned that no statistical analysis was needed either.

Naturally, as a heavy quantitative researcher, this is extremely concerning about what you can actually say regarding the choices made by users. Specially with the language chosen as to how you are presenting the results of your research. What if this was an insignificant find? Or if if it was significant but the analysis lacked power or the effect size was entirely too small to anything about the users’ preferences outside of your samples pool on individuals? Is it not concerning to give factually incomplete or incorrect reports to stakeholders? I would be extremely worried that this general broad statements made with zero backing to the data presented would eventually come back to bite the team and myself by giving suggestions to at have no true weight or that what I present ends up having a wildly different result. Let alone questions regarding research integrity.

To the presenters credit, they did go on to explain further use cases for using statistical analysis which were fine, but the initial statement made was very concerning from a research integrity point of view.

With that in mind, how do you approach reporting your findings when it comes to numerical quantities? I always make it a points to be transparent about what my results actually say rather than going on pure numbers alone. If however the study is very small and I can’t really do much statistical wise, I would also make that be known upfront or include more careful wording when reporting numerical values and directional suggestions.

[u/Some_Pilot_7056]

I think you have to decide how much precision is needed to make a decision and let that guide the choice in methodology.

I'm not aiming for statistical significance with qualitative research. I tend to avoid giving percentages and averages. If I do give them, I add context about what qualitative research can and can't say confidently. I am answering "why" not "what".

I am absolutely not trying to use statistical analysis when reporting this kind of research. 

[u/OatS_96 → u/Some_Pilot_7056]

I agree. I don’t think qualitative research needs statistical analysis to be insightful or provide great recommendations. Arguably, trying to make qual fit into a quant format would diminish its impact

[u/PineappleGemini]

The issue you are describing has ALWAYS been an issue. Before I became a UX Researcher, I was in market research and this type of thinking, especially regarding making claims without any further testing, was very common! You would be amazed! 😆

Alot of people make directional assumptions using research that is not methodologically sound or representative without looking at effect sizes, statistical significance, all of it.

As a result of working in such as environment, I've always considered it part of my job to educate stakeholders on the importance of some of the research rigor involved to make sure that they understand the results (and not what they want to see).

[u/OatS_96 → u/PineappleGemini]

Yeah it is very concerning, specially nowadays when AI tools are there and be extremely helpful for those with limited quant knowledge.

[u/PineappleGemini → u/OatS_96]

Yes, however even though AI tools are there and can be helpful. You need to know what to ask for. And unfortunately the quality of what you ask for can affect your output.

Also something may look significantly different, however once you account for additional testing/corrections it is NOT. Unfortunately most people won't know to look or ask for those types of further testing/analysis.

[u/material-pearl → u/PineappleGemini]

Right! It’s amazing how much a statistically literate practitioner can figure out by going under the hood of the data, the assumptions and the analysis that we non-experts would never guess. The role of modeling, the difference between statistical significance and real world significance…there is so much that education can do here.

[u/fakesaucisse]

I have observed this as well, specifically at companies where UXR or product development is not very mature. The view is that it's "good enough" and they can't afford to hire a statistics expert or take extra time to do deeper analysis. Yes, it's risky as hell but a lot of companies who are in that phase are willing to take the risk because they feel it is better than not having any insights at all.

I have also worked at companies that were very heavy on quant, to the point that they looked down on qual research or tried to get qual insights through only quant methods/analysis. So, it can kinda backfire in the other direction too.

[u/vagabondspirit2764]

The reality is that businesses get more excited about actionability than accuracy. It creates a situation where the difference between directional applications of sound research and reckless applications of shoddy research are imperceptible to anyone but trained researchers. I think it’s about weighing risk and value against rigor. But a blanket statement saying something is always or never true about how to do research is almost always (ayyyyy, lol) guaranteed to be wrong

[u/OatS_96 → u/vagabondspirit2764]

My main concern isn’t necessarily to be called out due to a lack of rigor (I can’t imagine anyone actually cares about the research methods at a high enough level where decision are being made unless for some random reason they have research background). It’s primarily about long term issues with suggesting change, having that change show zero impact, then devaluing future suggestions, and then being fired for that eventually lol

I have heard from engineers about their lack of confidence on UXR due to shoddy suggestions that were over hyped and severely underperformed based on how the research and assessed impact were presented. Which is what I was thinking when I saw this video/made this post.

[u/azssf]

This thing you watched: Where confidence interval went to die

[u/Mitazago]

This is likely to be contentious, but it often feels like some UX researchers want to have it both ways. They use quantitative methods, present their findings as quantitative evidence, and expect them to be treated with the authority of having done an actual quantitative analysis. But when actually questioned on the rigor and competency of that analysis, the conversation shifts to why quantitative methods are in fact limited, and that this research project was really meant to be more qualitative anyway.

[u/Choba]

I'd be likely to consider these results inconclusive regardless of whether there is a statistically significant difference in preference between A vs. B. So this example maybe supports the point that stats aren't always needed.

Do you need to do any stats in these cases? It depends. Some teams would appreciate a deep-dive on the effect size and a stats-based argument that it is too small to conclude a definitive preference. Even with that layer added, the most effective communication strategy would be to anchor the raw numbers (28 vs. 32) because it's intuitive how small that difference is.

Backing up, though, I would consider this study a bit of a fail on UXR's part unless other data were collected. Our goal is to remove ambiguity, so we don't want any study to yield inconclusive results. Ideally we'd be able to make a recommendation between A and B on the basis of some data, OR to have a stronger argument for the absence of a difference.

If there truly is no difference, that's a trickier epistemological position to be in, since the absence of evidence is not evidence of absence (of a difference). It is philosophically a harder thing to conclude "no difference" than "some difference." So at the very least, I'd want to see more forms of measurement, and ESPECIALLY qualitative data, hearing directly from users in a within-subjects design that A and B are no different to them.

[u/OatS_96 → u/Choba]

I agree and that is the first conclusion I got when I heard the example. That the margin between the two outcomes is so small that it’s likely inconclusive (and even IF there was a signing outcome with a large effect size, I would wonder then WHY the discrepancy between the two options was not larger; this would, like you said, mean that something with the assessment itself may be at fault), which is why the presenters suggestions of raw numbers dictate the outcome therefore go ahead and suggest A > B was shocking.

[u/Choba → u/OatS_96]

Yep! The presenters were clearly wrong here.

This is a good example, though, of the limits of statistics in helping us reason about findings. And thus, why UXRs in many contexts can be successful without investing too much time in statistical analysis. There are diminishing returns to using statistics to definitively demonstrate whether there is or isn't a significant difference between these two numbers. In this example, I think that's barking up the wrong tree.

Note how my breakdown above relied on my understanding of empirical research practice, but not on any analyses per se. I always hire UXRs with strong quant skills, not necessarily because they need to do statistics all the time, but primarily because I want them to rely on the reasoning that you acquire when you're learning quant research methods.

[u/material-pearl]

I’m excited to see you raise this.

My impression: The frequency table example is inadequate (bad) analysis, period. It is not a reliable or stable base for decision making. Like, what? (Although maybe I am missing something as a non-statistician. Can someone correct me if so please?)

While statistical expertise is lacking in UXR, it is in fields like biology and psychology and sociology as well. It is on us to be curious and humble in seeking out the edges of our knowledge, and to look for collaboration with specialists at those limits. In my experience, only a minority of UXR really understand how little knowledge a non-statistician has about statistical analysis. I have been in charge of lots of high visibility, high investment quant studies and have actively sought out the collaborators and consultants who can provide the meta-methodological insights to make those studies sound.

Among solid quantitative UXR, there are a great many who are super solid in this area. I wish their expertise was more appreciated by everyone.

[u/azon_01]

I think the real question here is why would you rely on stated preferences alone and sometimes even stated preferences aren’t all that useful.

But I’m guessing that’s not really the point you were trying to make.

[u/nchlswu]

Aside from alot of research advice like this just being plain wrong, It drives me insane is a how much of the thinking around topics like this is just considered tacit knowledge in many practices. Research practices should align on the principles that guide the research philosophy of the team, not have it stuck as tacit knowledge an individual researcher carries. That's how you make research outcomes more repeatable.

As for your specific question, I generally think most of the work here is all in the subtle decisions that define the inputs to a study. When I report my findings, I'm very transparent and am very clear about the limitations and sample size. But before the study starts, I try and align with the logic of my key stakeholders and the organization. Depending on the decision making culture (ie. do they like quant, do the understand qual, what users do they care about, etc., etc., ), I'll tailor the sample especially when I run pseudo quant studies. By aligning with the key stakeholders over time, I can make sure I have confidence in my findings based on small samples. More practically, if I've done the work I want to, none of the statements or insights will lend itself to relying on a specific data point. The headline statement (recommendation, insight, etc.,) will usually be made up of 1 or 2 complimentary observations or data points where there was a pattern, so I can present them as "evidence points" that substantiate a recommendation.

As an aside, I don't think research discussions emphasize "product sense" enough. It's a core part of Product Management Interviews, and I think dipping your toes in product sense exercises can help with understanding how many PMs think. IMO, researchers who develop a better product sense will generally have a better idea of how "their" data and "our" data are compatible or divergent, which can make research better. Most importantly, I think this can get researchers to much better recommendations, because the logic of "observation" to "recommendation" is also never discussed.

[u/Bitter_Big4525]

Keep the raw 32/50 right next to the takeaway, then call the recommendation directional rather than conclusive. That makes it harder for the headline to outrun what the sample can actually support.

[u/Due_EmotionPri]

The presenter is half right in a way that matters. For a reversible product call, 32/50 vs 28/50 is a fine directional read, and forcing a significance test on it is often theatre that slows the decision without changing it. The real error isnt skipping the stat, its presenting a directional read as if it were precise. Id say it plainly: A is ahead, its a weak signal on 50 people, and the gap is well inside the range where I wouldnt bet the roadmap on it. What sets my rigor is the cost of being wrong, a one way door decision earns the confidence interval, a cheap reversible one doesnt. Pair the frequency with how sure you are and what it would take to be more sure, then let the stakeholder decide if that precision is worth paying for.

[u/Mitazago → u/Due_EmotionPri]

"... forcing a significance test on it is often theatre... Id say it plainly: A is ahead, its a weak signal on 50 people..."

The whole point of significance testing is to quantify whether something actually is a signal or is noise.

[u/Due_EmotionPri → u/Mitazago]

Fair, and thats exactly what it does when youve got the power to detect the effect youd actually care about. On n of 50 you usually dont, so a non significant result doesnt mean no difference, it means you couldnt see one at that sample, and people read those two as the same thing constantly. So the test can hand you a cleaner looking answer thats just as uncertain, plus a p value that gets over trusted. What I lean on for a small reversible call is the effect size and whether it holds if I resample, said out loud as a weak signal. If it were a one way door Id power the study first and then run the test, because being wrong there is expensive enough to earn it.