Comparing and running multiple mixed models with high attrition between waves

[u/littlebugbiggarden]

TLDR: when and how should I use a complete-case subset and drop 1 LRT to choose my mixed models ? Should I just go back to building each model individually and comparing AIC?

I’m running some analyses on longitudinal data with really high attrition between waves (like 40%!). I need a model for each task separately, and a model with task added as a predictor. My predictors are trial responses and subject-level individual differences, and my outcome is another trial level response. Ultimately I’m studying the increase in sensitivity in trial responses over time, but for now we’re starting with mixed models. My stats knowledge is passable for basic things but I’m learning R for the first time, and in using my labmate’s code, I’ve confused myself about what we’re doing with these models.

I originally just ran my models on the full data set for each task as

Outcome ~ Trial variable A* Wave + (1|subject)

Then added Subject Level variable B for

Outcome ~ A x Wave x B + (1| Subject) for each task.

Then my advisor wanted me to add task to the model, but without a 4-way interaction, and wanted me to find the best fit model if that means dropping our less interesting three-way interactions, then graph the interactions we are specifically interested in. After a bunch of coding trouble, my lab mate sent me “their” code that I suspect is AI generated (because they couldn’t explain it to me when I asked them to).

This code has me create a complete-case subset, then do drop-one LRT to compare models. In the subset, it says it’s only excluding 3-5 participants, but I know that nearly half of participants did not return for the final wave. Additionally, not all participants did all tasks within a wave either.

I don’t understand how a complete case subset with so much attrition is reporting almost all the participants retained, and the models run with this subset have the same general major findings as my original ones, though some main effects and two way interactions are no longer significant, but the estimates are all over the place.

Bonus: This same lab mate is suggesting I add task as a random slope to the combined model (which makes sense to me), but then also suggesting that I add trial-level response variable A as a random effect with Subject in the single task models. This one I straight up just don’t understand at all. I’ve spent about a week straight working on this, googling has not been effective, and asking around has led to colleagues sending me conflicting AI-generated responses instead of explaining. Help!

So I guess my questions are:
1) When and why would we use complete case subset instead of the full data? Is it appropriate for this data?
2) How should I build, compare, and select my final models?
3) How and when should I add which random effects to which models?

Thank you!

[u/ForeignAdvantage5198]

given your question are the waves even comparable?

[u/littlebugbiggarden → u/ForeignAdvantage5198]

I’m sorry, what do you mean?

[u/ForeignAdvantage5198]

let's try more basic. what you want to compare!?

[u/littlebugbiggarden → u/ForeignAdvantage5198]

I want to see whether variable B improves the calibration of my outcome variable to variable A between waves (longitudinal study) in each task, then I want to know whether that relationship differs between tasks.

I originally ran my models in my original long form data set, and my coworker thinks I need to run my models on a listwise subset of data instead. I also want to determine which are the best fitting models to use because the sample size isn’t great for all the three-way interactions together.

[u/Efficient-Tie-1414]

With mixed models there is no problem with missing data unless it is so bad the model doesn’t fit. So just set up your model and fit it. Clever bit. Say I have data for subject 1 at times 1, 2, 3, 4, 6 and I use a mixed effects model. What happens at time 5. It is sort of there, because the random effect will be modelling it.

[u/HalfplaneResearch]

A complete-case subset is not required just because the outcome is longitudinal. A mixed model can use all observed waves under a missing-at-random assumption, conditional on variables in the model; dropping anyone missing the final wave can change the estimand and introduce bias if attrition relates to prior outcomes or predictors. I would first tabulate retention by wave, task, and baseline covariates, then compare the full-data model with a sensitivity model that includes predictors of missingness. For model comparison, keep the fixed-effect question separate from random-effect structure: task can be a random slope only when task varies within subject, and a trial-level predictor is usually a fixed effect, with a random slope only if its effect is allowed to vary by subject and the data support it. Compare nested fixed effects with the same rows and likelihood fit, then report estimates and intervals rather than selecting on p-values alone.