By Dr Arash Ardalan · · 4 min read
Key idea
Choose an analysis around the quantity you want to estimate and the way the data were collected. The same numerical measurements can need different analyses in different study designs.
Start with the question and outcome
Decide whether you want to estimate a difference, describe an association, predict an outcome, or assess change. Identify the outcome as continuous, binary, a count, ordinal, or a time-to-event measurement. Then identify the unit being studied: a person, a school, a business, or something else.
Write down the effect you want to report, such as a mean difference or a change in predicted probability. This makes the purpose of the analysis clear before software produces a p-value.
Match the method to the design
These are starting points. The distribution, sample size, measurement process and target effect determine which assumptions and alternatives need attention.
| Question and design | Possible starting point |
|---|---|
| Compare a continuous outcome in two independent groups | Welch’s two-sample t-test; regression when adjustment is needed |
| Measure a continuous outcome before and after in the same people | Paired t-test or a model for within-person change |
| Compare a continuous outcome across three or more independent groups | ANOVA, Welch’s ANOVA or a suitable regression model |
| Examine an association between categorical variables | Chi-squared test or an exact method when appropriate |
| Model a binary outcome with explanatory variables | Logistic regression |
| Analyse repeated observations or people clustered within groups | A mixed-effects model or another method that accounts for dependence |
Check independence and assumptions
Suppose 30 people each have a measurement before and after a course. The two observations from one person belong together. A paired analysis uses those within-person differences; treating all 60 observations as independent loses that structure.
Inspect plots and model diagnostics. For a paired t-test, the distribution of the differences is relevant. For a linear model, examine the residuals and the relationship being modelled. A single normality test should not determine the whole analysis.
Rank-based alternatives can be useful, but their interpretation depends on the method and assumptions. They do not automatically answer the same question about means or medians.
Report the result in context
- Report the effect estimate and a confidence interval, alongside a p-value where relevant.
- State the sample size, exclusions, missing-data approach and important model assumptions.
- Plan how multiple outcomes or comparisons will be handled.
- Interpret the size and uncertainty of the effect against the research question. A non-significant result does not establish that there is no effect.
Further reading
- R statistical documentation: independent and paired t-tests (opens in a new tab)
- NIST/SEMATECH: exploratory data analysis and statistical methods (opens in a new tab)
This guide provides general information. A study plan or business recommendation should reflect your own data, design and circumstances.
Back to all resources