Methods & Analysis Review
Voluntary participation & a 12-week outcome.
Illustrative reviewer question How was the adjustment set chosen, and what can the adjusted comparison establish about voluntary participation and the 12-week outcome?
Overall assessment
Document the design before interpreting the adjusted result.
The reported difference falls from +5.55 to +0.96 outcome-score points after adjustment. The supplied outputs support a substantially smaller adjusted association. They do not, by themselves, establish that participation caused the difference.
Next step. Document the adjustment rationale and measurement timing, align the claim with the study design, and present uncertainty and practical meaning alongside the estimate.
Materials reviewed. Synthetic study summary, stated adjustment specification, results table, and sensitivity chart. The source example describes 1,200 records across four synthetic sites, with complete 12-week outcomes by construction.
Outside this review. Underlying records, data provenance, executable code, model diagnostics, and independent reproduction. No new analyses were performed for these review notes.
01 Material · Study design
Document why the adjustment supports the intended claim.
- Evidence
- The reported difference falls from +5.55 points unadjusted to +0.96 after adjustment for baseline score, need, age, and site. The supplied summary names these variables but does not explain their selection or fully document when they were measured relative to participation. It labels the adjustment as prespecified without providing a dated analysis plan.
- Why it matters
- The main estimate changes substantially with adjustment. Readers need the rationale and measurement timing to assess whether the model addresses the intended question. The smaller estimate alone does not establish that confounding has been removed or that participation caused the difference.
- Action
- State whether the question concerns association or causation. Explain the adjustment set and measurement timing, and identify the record supporting prespecification. If a causal interpretation is intended, document the identification assumptions and their justification; otherwise, keep the interpretation associational.
02 Should fix · Uncertainty
Show the full sensitivity range without a pass/fail threshold.
- Evidence
- The flexible adjustment gives +0.95 points, close to the primary +0.96. Omitting one site at a time produces estimates from +0.66 to +1.24. The intervals include zero when site B or D is omitted.
- Why it matters
- These estimates remain positive, and their intervals overlap substantially. A change in whether an interval includes zero does not establish that sites differ or that the underlying association disappears. The checks describe uncertainty across the supplied specifications; they are not independent replications.
- Action
- Present the primary estimate alongside all supplied sensitivity results. Explain what each check changes and retain its interval. Avoid selecting the specification that gives the strongest result or describing zero-crossing alone as proof of instability.
Inspect all seven supplied estimates
| Specification | Estimate | 95% interval |
|---|---|---|
| Unadjusted | +5.55 | +4.47 to +6.63 |
| Primary adjustment | +0.96 | +0.09 to +1.82 |
| Flexible adjustment | +0.95 | +0.09 to +1.82 |
| Leave out site A | +1.10 | +0.05 to +2.14 |
| Leave out site B | +0.89 | −0.09 to +1.88 |
| Leave out site C | +1.24 | +0.23 to +2.25 |
| Leave out site D | +0.66 | −0.30 to +1.62 |
Primary adjustment: baseline score, need, age, and site. Flexible adjustment adds baseline², need², and baseline × need. Each site check omits one synthetic site and refits the primary specification. Reported intervals use HC1 standard errors.
View the source example and sensitivity chart03 Should fix · Decision context
Explain what a one-point difference means.
- Evidence
- The adjusted estimate is +0.96 outcome-score points. The reviewed summary does not define the score’s practical interpretation or a meaningful-difference benchmark.
- Why it matters
- A numerical difference can be estimated precisely yet have little practical value. Readers cannot assess the importance of this result from its sign or interval alone.
- Action
- Document the score’s range and interpretation. Where a defensible external benchmark exists, explain its relevance and compare it with the estimate and uncertainty. If no benchmark exists, state that limitation; do not invent one to make the result look favorable.
From notes to decisions
Points your response to reviewers should address.
Use these questions to guide clarification by email and your own revisions. They are a checklist, not submission-ready text.
- What is the intended claim, and how do the design, adjustment set, and measurement timing support it?
- How do the primary result, sensitivity checks, and uncertainty affect that claim?
- What does a one-point difference mean in the study’s context?
- For an actual study: how many participants were eligible, enrolled, followed up, and analysed in each group? Report missing outcomes and exclusion reasons, explain how they were handled, and state when follow-up is complete.
End of sample · Actual findings depend on your materials and agreed scope.