What can (and cannot) a model say about a psychological construct?
Table of contents
A model that predicts well does not necessarily measure well. Before interpreting a score, ask about its validity.
Predicting is not measuring
Scale reliability is often summarized with Cronbach’s alpha:
High reliability does not guarantee the instrument measures the construct we think it does.
Minimal example
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(scores.mean())
AUC tells you how well the classes separate, not whether the variable makes clinical sense.
Questions before trusting the model
- Is the instrument validated in this population?
- Is performance similar across groups?
- Who benefits, and who could be harmed by a wrong prediction?