MAISON-LLF: the data, the forecasting paper, and what actually wins

The shape of one patient

~56 days of daily sensor rows, punctuated by ~4 clinical visits where the questionnaires are filled in. Everything between visits is sensor data only.

Why this size matters

• Daily aggregates, not raw traces. The watch samples every second, but you receive one row per patient per day (step totals, mean HR, sleep-stage minutes…).
• Sparse labels. Questionnaires arrive only every ~2 weeks, so each patient yields just ~4 supervised targets per scale.
• The unit of evidence is the patient, not the row. ~18 independent people — tiny by deep-learning standards, normal by clinical-trial standards. Any claim must survive at this N.
• Held-out by subject. Splits are patient-level so a model is judged on people it has never seen — the realistic deployment condition.

Oxford Hip / Knee Score

12 items · each 0–4 · total 0–48 · higher = better joint function / less pain. Hip patients get OHS, knee patients get OKS (structurally identical).

e.g. “Climbing stairs is painful” → 0 (always) … 4 (never)

Social Isolation Score

6 items · each 1–5 · total 6–30 · higher = less isolated.

e.g. “I have people I can talk to” → 1 (never) … 5 (always)

Physical performance

tug Timed-Up-and-Go (seconds, lower = better) · chairstand sit-stands in 30 s (count, higher = stronger). One continuous value each.

Channel count by group

Between- vs within-patient spread

Standard deviation of patient means (between) vs mean of within-patient SD. Between dominates on every scale.

The 3 audit heuristics

• A. Long constant run — flag_long_run: longest identical run ≥ 14 days.
• B. Low unique count — flag_low_unique: ≤ 5 distinct values across all 56 days.
• C. Zero variance — flag_zero_sd: σ < 1e-6 (channel is a constant).

A cell tripping any flag is treated as imputed. Cross-checked on high-trust patients to keep false positives low.

Per-subject trust score

% of a patient's 46 channels passing all 3 audits. Hover a bar. Two training subjects (8, 18) are >70% imputed — a sensor model is largely fitting constants for them.

The problem with the prior framing

Earlier MAISON-LLF analyses attached each clinical label to the sensor window preceding or around the assessment. That blurs into concurrent estimation and risks information leakage — using data from on/after the assessment day to “predict” it.

In deployment you must predict before the visit, from past data only.

The leakage-safe reformulation

Predict the score at assessment n using only observations up to day T_n − 7 (a 7-day-ahead forecast). Histories are expanding windows: the first visit forecasts from the first ~7 days, the second from ~21 days, and so on.

Strict chronological input→target separation, patient-level test split.

MAE — Mean Absolute Error

MAE = mean(|predicted − actual|). The average size of the miss, in the score's own units. Lower is better; 0 is perfect. Easy to read: “OHS total MAE 5.23” = off by ~5.2 points on a 0–48 scale on average.

Robust to outliers (no squaring). But it has no built-in reference — “5.2” is only good or bad relative to a baseline, which is the whole §10 argument.

RMSE — Root Mean Squared Error

RMSE = sqrt(mean((predicted − actual)²)). Like MAE but squares errors first, so big misses are punished much more. Lower is better. RMSE ≥ MAE always; a large gap between them signals a few large errors (here: one atypical test patient).

R² — coefficient of determination

R² = 1 − SS_res / SS_tot = the fraction of variance the model explains relative to just predicting the mean. 1 = perfect; 0 = no better than predicting the mean; <0 = worse than the mean.

Pearson r — correlation

r ∈ [−1, 1]: do predictions and truth move together, regardless of offset or scale? +1 = perfect ranking, 0 = none, −1 = inverted.

r ignores systematic bias — you can have decent r and terrible R² if predictions track the trend but sit consistently too high/low. That is exactly the paper's OHS total: r=+0.53 but R²=−1.06 (right order, wrong level). And sis-02 has r=−0.79: predictions move the wrong way.

Full metric table (test, multi-task)

Total-score r > per-item r: summing 12/6 items cancels item noise and lets the coarse recovery trend show. R² stays negative — predictions track order but carry a systematic offset.

Per-item Pearson r — very uneven

Hover bars. A few items are learnable (ohs-12: r=0.84, the only positive-R² item); some are anti-correlated (sis-02: r=−0.79 — sensors mislead for that social item).

The few that work, the many that don't

• ohs-12 is the one real success — the only item with positive R² (+0.353) and a strong r (+0.839). Some recovery dimension it captures is consistently reflected in the sensors.
• ohs-07, ohs-08, ohs-09 have positive r but negative R² — the model tracks their trend but with a systematic offset (right direction, wrong level).
• ohs-11 is the worst (MAE 2.15, R²=−3.94) — and recall from §2 it carries the most total-score variance, so the model fails hardest exactly where it matters most.
• sis-02 is actively misleading — r=−0.792, predictions anti-correlate with truth. The sensor cues informative for other items point the wrong way for this social item, which has no clear behavioural correlate in a wearable.

Why per-item is so much weaker than totals

Items are quantised to 5 levels and individually noisy; the network hedges toward the middle. Only when 12/6 items are summed do independent errors cancel and the coarse recovery trend emerge — lifting total-score r well above per-item r.

No cross-validation — a single 3-patient test set

The model is evaluated on one fixed split (12 train / 3 val / 3 test). It does not use leave-one-subject-out or patient-grouped k-fold CV. With only 3 test patients (12 events), every reported number is one draw from a very high-variance distribution — and the residual plots show a single atypical patient (P2) dominates the metrics. A different 3-patient draw could easily flip “multi-task wins”.

SHAP feature-selection leakage (caught in review)

Reviewer 2 noticed the central “leakage-safe” claim could be undermined by how features were chosen. The authors disclosed that in the original submission SHAP was computed on the full dataset — letting test-set information steer feature selection. The camera-ready recomputes SHAP on the training set only.

They declined per-fold SHAP (which pairs naturally with k-fold CV), arguing it yields different rankings per fold and needs fold-specific pipelines; instead they use one fixed training-only ranking. Defensible for simplicity, but it is also the reason the work stays on a single split rather than CV — the two limitations are linked.

Negative R² everywhere

Almost all R² values are below 0 — i.e. worse than predicting the cohort mean on variance-explained. Only ohs-12 is positive. The model captures order (positive r on some targets) but not level.

Beaten by persistence

As §12–13 show, the trivial “predict last/average score” baselines beat the GRU on the same metrics. The paper does not report total-score-vs-persistence; our companion work supplies it.

Conclusion overstated (caught)

The original conclusion sold “~one-point average error” as forecasting ability. Reviewer 2 noted that is ~25% of the Likert range, with R²<0 and a losing comparison to persistence. The authors agreed and revised to frame the work as a leakage-safe formulation, not reliable prediction.

① Variance budget = a hard ceiling

Shrout–Fleiss ICC + cluster bootstrap: 66–91% of score variance is between-patient. A patient-blind model can address at most a few % — persistence captures the rest for free.

② Sub-item battery can shrink

Pure psychometrics (Cronbach α + greedy selection + bootstrap stability): SIS 6→5 robustly, OHS 12→8 mostly, OKS does not shrink on this cohort. ~33% less patient burden, defensible.

③ Upstream-imputation discovery

0 NaN → ~17% of cells are filled constants; 2 training subjects <30% trusted. We ship a per-cell trust mask + a re-runnable clean pipeline.

④ Persistence ceiling holds

On clean data, persistence is unbeaten by all 8 pre-registered univariate tests and by a multivariate ridge across 5 targets (0/5 beat it; mean Δ = +0.0046 NMAE, i.e. worse).