What can (and cannot) a model say about a psychological construct?

What can (and cannot) a model say about a psychological construct?

Esteban Navarro Díaz

Table of contents

A model that predicts well does not necessarily measure well. Before interpreting a score, ask about its validity.

Predicting is not measuring

Scale reliability is often summarized with Cronbach’s alpha:

α=kk−1(1−∑i=1kσi2σX2)\alpha = \frac{k}{k-1}\left(1 - \frac{\sum_{i=1}^{k}\sigma_i^2}{\sigma_X^2}\right)

High reliability does not guarantee the instrument measures the construct we think it does.

Minimal example

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

model = LogisticRegression(max_iter=1000)
scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc")
print(scores.mean())

AUC tells you how well the classes separate, not whether the variable makes clinical sense.

Questions before trusting the model

  1. Is the instrument validated in this population?
  2. Is performance similar across groups?
  3. Who benefits, and who could be harmed by a wrong prediction?