ANOVA, or analysis of variance, is a family of statistical tests used to determine whether group means differ beyond what random variation would reasonably explain. It compares variation between group means with variation within groups and produces an F statistic. A significant result indicates that at least one mean differs, but not which mean is different.
ANOVA is one of the most widely taught methods for comparing groups in research. It is used in fields such as psychology, education, medicine, agriculture, business, engineering, biology and the social sciences.
This guide explains what ANOVA means, how it works, which type to choose, how to check its assumptions, how to interpret an ANOVA table and how to report the findings. It also explains common alternatives, including Welch ANOVA, the Kruskal–Wallis test and mixed-effects models.
Key takeaways
- ANOVA compares group means by analysing between-group and within-group variation.
- A significant omnibus ANOVA shows that not all population means are equal; it does not identify the differing groups.
- The correct ANOVA depends on the research design, especially the number of factors and whether observations are independent or repeated.
- Classical between-groups ANOVA assumes independent observations, approximately normal residuals and equal error variances.
- Effect sizes, confidence intervals, descriptive statistics and adjusted follow-up comparisons should accompany the p-value.
- Welch ANOVA is often preferable when independent groups have unequal variances.
What is ANOVA?
Analysis of variance is a statistical modelling framework used to test whether the expected values, usually means, of two or more groups or experimental conditions are equal. It estimates how much observed variation is associated with the factor being studied and compares it with unexplained or residual variation.
Although ANOVA is most commonly introduced as a test for three or more groups, it can also be applied to two groups. With two independent groups and matching assumptions, a one-way ANOVA produces an F statistic equivalent to the square of the pooled independent-samples t statistic.
Researchers usually prefer a t-test for a simple two-group comparison because its interpretation is more direct. ANOVA becomes especially useful when there are three or more groups, multiple factors, repeated measurements, blocking variables or interaction effects.
Why does ANOVA analyse variance to compare means?
The name can initially seem confusing. ANOVA is normally used to test mean differences, yet it does so by comparing two estimates of variability:
- Between-group variation: variation associated with differences among the group means.
- Within-group variation: variation among observations belonging to the same group.
When the population means are equal, the between-group and within-group mean squares should estimate approximately the same error variance. Their ratio should therefore be close to one.
When the group means are separated relative to the amount of variation within groups, the between-group mean square becomes larger. This produces a larger F statistic and stronger evidence against the equal-means null hypothesis.
ANOVA can therefore be understood as a signal-to-noise comparison:
[
F=\frac{\text{variation explained by the factor}}{\text{unexplained or residual variation}}
]
When should ANOVA be used?
ANOVA is appropriate when:
- The outcome or dependent variable is quantitative.
- One or more explanatory variables are categorical factors.
- The research question concerns differences in expected outcomes across factor levels.
- The study design and data structure are represented by the chosen ANOVA model.
- Observations are independent unless the model explicitly handles repeated or clustered observations.
- The residual assumptions are sufficiently reasonable for the intended inference.
Example research questions
- Do three teaching methods produce different mean examination scores?
- Does crop yield differ among four fertilisers?
- Do treatment type and sex jointly affect mean recovery time?
- Does mean anxiety change across three measurement occasions?
- Does the effect of a learning programme differ by age group?
- Are average manufacturing measurements different across production lines?
When ordinary ANOVA may be inappropriate
A standard between-groups ANOVA should not automatically be used when:
- The outcome is binary, a count, a proportion with a restricted denominator or a time-to-event measure.
- The same participant or unit contributes multiple observations but dependence is ignored.
- Participants are clustered in schools, hospitals, families or other groups.
- The outcome has severe skew, influential outliers or a distribution incompatible with a normal-error model.
- The variances are strongly unequal and group sizes are unbalanced.
- The research question concerns several dependent variables jointly.
- Important continuous covariates need to be controlled.
- The design contains random, nested, split-plot or crossed effects that require a more specialised model.
Depending on the problem, a generalised linear model, mixed-effects model, ANCOVA, MANOVA, survival model, robust procedure or rank-based method may be more suitable.
Essential ANOVA terminology
Factor
A factor is a categorical explanatory variable. Examples include treatment, school type, teaching method or production line.
Factor level
A level is one category of a factor. If the factor is teaching method, its levels might be lecture, blended learning and problem-based learning.
Response variable
The response, outcome or dependent variable is the quantitative measurement being analysed, such as test score, blood pressure or crop yield.
Main effect
A main effect is the average association between one factor and the outcome after accounting for the other terms included in the model.
Interaction effect
An interaction occurs when the association between one factor and the outcome depends on the level of another factor.
For example, a tutoring programme may improve scores for beginning students but have little effect for advanced students. In that case, the effect of tutoring interacts with prior skill level.
Residual
A residual is the difference between an observed value and the value predicted by the fitted model. Many ANOVA assumptions concern the residuals rather than the raw outcome considered without its group structure.
Omnibus test
ANOVA is an omnibus test because it evaluates an overall hypothesis about a set of means or model terms. A significant omnibus test does not by itself identify the specific means that differ.
Types of ANOVA
| Type | Design | Main question | Example |
|---|---|---|---|
| One-way between-groups ANOVA | One categorical factor; different units in each group | Do the group means differ? | Do three teaching methods produce different mean scores? |
| Two-way or factorial ANOVA | Two categorical factors | Are there main effects and an interaction? | Do teaching method, school type and their combination affect scores? |
| Repeated-measures ANOVA | The same units measured across conditions or times | Does the mean change within the same units? | Does anxiety change before, during and after treatment? |
| Mixed ANOVA | At least one between-units factor and one within-units factor | Do groups change differently over time or conditions? | Do treatment and control groups improve differently over three visits? |
| Welch one-way ANOVA | Independent groups with unequal variances | Do means differ without assuming equal variances? | Do mean incomes differ across sectors with different variability? |
| Randomised-block ANOVA | Treatments compared within relatively homogeneous blocks | Do treatment means differ after accounting for blocks? | Compare fertilisers while blocking by field location. |
| Nested ANOVA | Levels of one factor occur only within levels of another | How is variation distributed across nested sources? | Pupils nested within classrooms nested within schools. |
One-way ANOVA
A one-way ANOVA includes one categorical factor and one quantitative outcome. It is the standard method for comparing several independent group means under a common-variance model.
Two-way and factorial ANOVA
A two-way ANOVA includes two factors. A factorial ANOVA can include two or more factors.
A factorial design evaluates:
- The main effect of each factor.
- Interactions among factors.
- Residual variation not explained by the model.
Factorial designs can be more informative than running separate one-way tests because they can reveal interactions and estimate effects within a unified model.
Repeated-measures ANOVA
Repeated-measures ANOVA is used when the same subjects or experimental units are measured under multiple conditions or at several times.
Measurements from the same unit are correlated. A repeated-measures analysis models this dependence rather than treating every measurement as independent.
For within-subject factors with more than two levels, classical univariate repeated-measures ANOVA commonly requires a covariance condition called sphericity. When sphericity is not reasonable, corrections such as Greenhouse–Geisser may be applied, or a mixed-effects model may be preferable.
Mixed ANOVA
A mixed ANOVA combines:
- A between-subjects factor, such as treatment group.
- A within-subjects factor, such as time.
Its most important question is often whether the groups change differently over time. This is represented by the group-by-time interaction.
Welch ANOVA
Welch ANOVA tests mean differences across independent groups without assuming a common population variance. It uses variance-based weights and adjusted degrees of freedom.
It is particularly useful when:
- Group variances differ.
- Group sizes are unequal.
- The largest variance is associated with a small group or the smallest variance with a large group.
Welch’s method is a principled alternative to forcing a classical equal-variance ANOVA onto heterogeneous data (Welch, 1951).
Related methods: ANCOVA and MANOVA
ANCOVA, or analysis of covariance, extends ANOVA by including one or more continuous covariates.
MANOVA, or multivariate analysis of variance, considers several dependent variables jointly.
These methods require their own assumptions and should not be treated as interchangeable labels for ordinary ANOVA.
The hypotheses tested by ANOVA
For a one-way ANOVA with (k) population means, the null hypothesis is:
[
H_0:\mu_1=\mu_2=\cdots=\mu_k
]
The alternative hypothesis is:
[
H_A:\text{not all population means are equal}
]
The alternative does not state that every mean differs from every other mean. It only states that at least one equality in the null hypothesis does not hold.
For factorial ANOVA, separate hypotheses are tested for each main effect and interaction.
How ANOVA works
Consider observation (y_{ij}), where (i) represents the group and (j) represents an observation within that group.
Total sum of squares
The total sum of squares measures the overall variation around the grand mean:
[
SS_{\text{Total}}=\sum_i\sum_j(y_{ij}-\bar y_{\cdot\cdot})^2
]
Between-group sum of squares
The between-group sum of squares measures variation associated with differences between group means and the grand mean:
[
SS_{\text{Between}}=\sum_i n_i(\bar y_{i\cdot}-\bar y_{\cdot\cdot})^2
]
Within-group sum of squares
The within-group or error sum of squares measures variation of observations around their own group means:
[
SS_{\text{Within}}=\sum_i\sum_j(y_{ij}-\bar y_{i\cdot})^2
]
For a standard one-way ANOVA:
[
SS_{\text{Total}}=SS_{\text{Between}}+SS_{\text{Within}}
]
This decomposition is the central organising principle of the one-way ANOVA table (National Institute of Standards and Technology, n.d.).
Degrees of freedom
If there are (k) groups and (N) total observations:
[
df_{\text{Between}}=k-1
]
[
df_{\text{Within}}=N-k
]
[
df_{\text{Total}}=N-1
]
Mean squares
[
MS_{\text{Between}}=
\frac{SS_{\text{Between}}}{df_{\text{Between}}}
]
[
MS_{\text{Within}}=
\frac{SS_{\text{Within}}}{df_{\text{Within}}}
]
F statistic
[
F=
\frac{MS_{\text{Between}}}{MS_{\text{Within}}}
]
The p-value is calculated from an F distribution with the numerator and denominator degrees of freedom associated with the model term and error term.
How to read an ANOVA table
A standard one-way ANOVA table has the following structure:
| Source | Sum of squares | Degrees of freedom | Mean square | F | p |
|---|---|---|---|---|---|
| Between groups | (SS_B) | (k-1) | (SS_B/(k-1)) | (MS_B/MS_W) | F-distribution probability |
| Within groups/error | (SS_W) | (N-k) | (SS_W/(N-k)) | — | — |
| Total | (SS_T) | (N-1) | — | — | — |
To interpret it:
- Identify the tested factor or model term.
- Check its degrees of freedom.
- Read the F statistic.
- Read the p-value.
- Examine the effect size and confidence interval.
- Inspect descriptive statistics and plots.
- If appropriate, interpret planned contrasts or multiplicity-adjusted comparisons.
- Confirm that the model and diagnostic checks support the interpretation.
A p-value should not be interpreted as the probability that the null hypothesis is true. It also does not indicate the size or practical importance of an effect. Statistical evidence should be evaluated together with study design, effect estimates, uncertainty and subject-matter knowledge (Wasserstein & Lazar, 2016).
Worked one-way ANOVA example
Suppose a researcher compares examination scores from three teaching methods.
| Method A | Method B | Method C |
|---|---|---|
| 72 | 80 | 77 |
| 75 | 82 | 76 |
| 78 | 79 | 79 |
| 74 | 83 | 75 |
| 76 | 81 | 78 |
The group means are:
- Method A: 75
- Method B: 81
- Method C: 77
The grand mean is:
[
\bar y_{\cdot\cdot}=77.67
]
Step 1: Calculate between-group variation
[
SS_{\text{Between}}=93.33
]
Step 2: Calculate within-group variation
[
SS_{\text{Within}}=40.00
]
Step 3: Confirm total variation
[
SS_{\text{Total}}=93.33+40.00=133.33
]
Step 4: Calculate degrees of freedom
There are three groups and 15 observations:
[
df_{\text{Between}}=3-1=2
]
[
df_{\text{Within}}=15-3=12
]
[
df_{\text{Total}}=15-1=14
]
Step 5: Calculate mean squares
[
MS_{\text{Between}}=\frac{93.33}{2}=46.67
]
[
MS_{\text{Within}}=\frac{40.00}{12}=3.33
]
Step 6: Calculate the F statistic
[
F=\frac{46.67}{3.33}=14.00
]
The completed table is:
| Source | SS | df | MS | F | p |
|---|---|---|---|---|---|
| Teaching method | 93.33 | 2 | 46.67 | 14.00 | < .001 |
| Error | 40.00 | 12 | 3.33 | — | — |
| Total | 133.33 | 14 | — | — | — |
Step 7: Interpret the omnibus test
The result is:
[
F(2,12)=14.00,\ p<.001
]
The data provide evidence against the hypothesis that all three population means are equal.
The omnibus result does not identify the differing methods.
Step 8: Conduct adjusted follow-up comparisons
Tukey-adjusted comparisons show:
- Method B is higher than Method A by 6 points, adjusted (p<.001).
- Method B is higher than Method C by 4 points, adjusted (p=.012).
- Methods A and C do not show a statistically significant difference, adjusted (p=.234).
Step 9: Calculate effect sizes
For this example:
[
\eta^2=
\frac{SS_{\text{Between}}}{SS_{\text{Total}}}
=\frac{93.33}{133.33}=.70
]
A bias-adjusted omega-squared estimate is approximately:
[
\omega^2=.63
]
This deliberately simple dataset was constructed to demonstrate the calculation and contains a very large difference. It should not be treated as evidence about real teaching methods.
ANOVA assumptions
1. Independence of observations
Observations must be independent unless the selected model explicitly represents repeated, paired, clustered or hierarchical data.
Independence is determined mainly by the research design and data-collection process. A software test cannot prove that observations are independent.
Examples of violations include:
- Measuring the same participant several times and treating all measurements as unrelated.
- Analysing pupils from the same classroom without accounting for clustering.
- Using several measurements from the same specimen as though they were independent experimental units.
- Allowing participants in one condition to influence participants in another condition.
When observations are dependent, repeated-measures, multilevel, mixed-effects or other correlation-aware models may be needed.
2. Approximately normal residuals
Classical F-test calculations are based on a normal-error model. The relevant distribution is the distribution of residuals within the model structure, not necessarily the pooled raw outcome.
Check normality using:
- Residual Q–Q plots.
- Histograms or density plots of residuals.
- Group-specific plots.
- Investigation of skew and influential observations.
- Subject-matter knowledge about the measurement process.
A normality test should not be used as an automatic pass-or-fail switch. With small samples it may have little power, while with very large samples it may detect unimportant departures.
3. Homogeneity of variance
Classical between-groups ANOVA assumes that the error variance is equal across groups or model cells.
Useful checks include:
- Group standard deviations.
- Box plots.
- Residual-versus-fitted plots.
- Levene or Brown–Forsythe variance tests.
- Consideration of whether the measurement scale naturally produces larger variation at larger means.
When independent groups have unequal variances, especially with unequal sample sizes, Welch ANOVA is often more defensible than the pooled-variance procedure.
4. Absence of highly influential observations
ANOVA is based on squared deviations, so extreme observations can strongly influence means, sums of squares and F statistics.
Outliers should be investigated rather than automatically deleted. Researchers should check:
- Whether the value is a data-entry or measurement error.
- Whether it belongs to the intended population.
- Whether the analysis changes materially with and without it.
- Whether a transformation, robust method or alternative distribution is justified.
Any exclusions or sensitivity analyses should be documented transparently.
5. Correct model specification
A technically correct calculation can still answer the wrong question. The model must represent:
- The experimental unit.
- The assignment or sampling process.
- Repeated measurements.
- Blocks and clusters.
- Relevant factors.
- Interactions.
- Covariates.
- The outcome’s measurement scale.
What should you do when assumptions are violated?
| Problem | Possible response |
|---|---|
| Unequal variances across independent groups | Use Welch ANOVA and consider Games–Howell comparisons. |
| Strong skew with a meaningful positive scale | Consider a justified transformation or a model with a more suitable outcome distribution. |
| Influential outliers | Verify data, report sensitivity analyses, or use a robust method. |
| Repeated measurements | Use repeated-measures ANOVA or a mixed-effects model. |
| Clustered observations | Use a multilevel, mixed-effects or cluster-aware model. |
| Sphericity violation | Use an appropriate correction or fit a model that represents the covariance structure. |
| Binary outcome | Use logistic regression or another binomial model. |
| Count outcome | Consider Poisson or negative-binomial modelling. |
| Ordinal or highly non-normal outcome | Consider an ordinal, robust, permutation or rank-based method, depending on the research question. |
| Several dependent variables | Consider MANOVA or separate outcome models with multiplicity control. |
| Important continuous covariates | Consider ANCOVA or regression. |
Is Kruskal–Wallis a nonparametric ANOVA?
The Kruskal–Wallis test is commonly described as a nonparametric alternative to one-way ANOVA, but the two procedures do not test exactly the same parameter.
ANOVA tests a model-based hypothesis about means. Kruskal–Wallis uses ranks and tests whether group distributions differ in location under suitable conditions. It should not automatically be described as a test of medians unless additional distributional assumptions support that interpretation.
Post-hoc tests and planned contrasts
Why follow-up analysis is needed
A significant omnibus ANOVA tells you that the full equal-means hypothesis is not compatible with the data at the selected significance level. It does not reveal which means differ.
Follow-up methods should match the research question and model assumptions.
Tukey HSD
Tukey’s honestly significant difference procedure is designed for all pairwise comparisons among means under the common-variance ANOVA model. It controls the familywise error rate across the set of pairwise tests (Tukey, 1949).
Use it when:
- All pairwise comparisons are scientifically relevant.
- The classical equal-variance model is reasonable.
- The comparisons were not limited to a specific control or prespecified set.
Games–Howell
Games–Howell is commonly used for pairwise comparisons when variances or sample sizes are unequal. It does not pool all groups into a single common error variance in the same way as Tukey HSD (Games & Howell, 1976).
It is a natural follow-up to Welch one-way ANOVA.
Dunnett comparisons
Dunnett-type procedures are appropriate when several treatments are compared with one control and comparisons among all treatment pairs are not required.
Holm and Bonferroni adjustments
Holm or Bonferroni corrections can be applied to a defined family of comparisons. Holm’s sequential method is generally less conservative than a simple Bonferroni correction while controlling the familywise error rate.
Planned contrasts
A planned contrast tests a scientifically specified comparison, such as:
- Control versus the average of two treatments.
- A linear dose trend.
- Two active treatments versus placebo.
- One theory-driven combination of factor levels.
Prespecified contrasts may be more informative and more powerful than conducting every possible pairwise comparison. The analysis plan should state the contrast, coding and multiplicity strategy.
A ritualistic rule that no contrast may ever be examined unless the omnibus test is significant is too simplistic. The validity of a prespecified contrast depends on the planned hypothesis family and error-control strategy.
Effect sizes for ANOVA
A p-value addresses compatibility with a statistical null model. It does not show how large or important a difference is.
Eta squared
For a one-way ANOVA:
[
\eta^2=
\frac{SS_{\text{Between}}}{SS_{\text{Total}}}
]
Eta squared describes the proportion of sample variation associated with the factor in that model. It can overestimate the corresponding population effect.
Partial eta squared
For a model term:
[
\eta_p^2=
\frac{SS_{\text{Effect}}}
{SS_{\text{Effect}}+SS_{\text{Error}}}
]
Partial eta squared is frequently reported in factorial and repeated-measures analyses. Its value depends on which other sources of variation are excluded from the denominator, so comparisons across different designs can be misleading.
Omega squared
For a one-way fixed-effects design, omega squared can be estimated as:
[
\omega^2=
\frac{SS_{\text{Between}}-(k-1)MS_{\text{Within}}}
{SS_{\text{Total}}+MS_{\text{Within}}}
]
Omega squared adjusts for some upward bias in eta squared and is often useful as a population-oriented estimate.
Cohen’s f
Cohen’s (f) is related to eta squared:
[
f=
\sqrt{\frac{\eta^2}{1-\eta^2}}
]
It is frequently used in power and sample-size calculations.
Reporting effect sizes responsibly
Researchers should:
- Name the exact effect-size measure.
- Avoid writing “eta squared” when the software produced partial eta squared.
- Provide confidence intervals when available.
- Interpret magnitude in the study’s substantive units and context.
- Avoid treating generic small, medium and large cut-offs as universal laws.
- Report pairwise mean differences and their confidence intervals when specific comparisons matter.
Effect-size reporting improves the value of findings for replication, power analysis and meta-analysis (Lakens, 2013).
Main effects and interactions
In a factorial ANOVA, the effect of one factor may depend on another factor.
Suppose a study tests:
- Teaching method: lecture or blended.
- Prior achievement: lower or higher.
- Outcome: examination score.
A significant teaching-method-by-achievement interaction means that the teaching-method difference is not constant across achievement groups.
How to interpret an interaction
- Plot the cell means and confidence intervals.
- Identify the substantive pattern.
- Estimate simple effects or relevant contrasts.
- Use multiplicity adjustments appropriate to the planned comparison family.
- Report the direction and magnitude of the interaction.
- Avoid interpreting marginal main effects as though the interaction did not exist.
A non-significant interaction does not prove that the factor effects are identical. The confidence interval around the interaction estimate should also be considered.
ANOVA compared with related tests
| Method | Typical use | Important distinction |
|---|---|---|
| Independent-samples t-test | Compare two independent means | Under equal-variance assumptions, equivalent to a two-group one-way ANOVA. |
| Paired t-test | Compare two related means | Represents paired or repeated observations. |
| One-way ANOVA | Compare means across levels of one factor | Classical version assumes a common error variance. |
| Welch ANOVA | Compare independent means with unequal variances | Uses adjusted weighting and degrees of freedom. |
| Factorial ANOVA | Estimate main effects and interactions | Includes two or more categorical factors. |
| Repeated-measures ANOVA | Compare related means across conditions or times | Models within-unit dependence and may require sphericity. |
| ANCOVA | Compare adjusted means while including covariates | Requires careful interpretation of covariate relationships and interactions. |
| MANOVA | Analyse several outcomes jointly | Tests multivariate mean patterns rather than one outcome. |
| Kruskal–Wallis test | Rank-based comparison of independent groups | Does not directly test the ordinary mean-equality hypothesis. |
| Linear regression | Model a quantitative outcome using coded predictors | ANOVA with categorical predictors is a form of linear modelling. |
| Mixed-effects model | Analyse clustered, repeated or hierarchical data | Adds random effects and flexible covariance structures. |
ANOVA and regression
ANOVA and ordinary linear regression are not competing mathematical systems. Standard fixed-effects ANOVA can be represented as a linear model in which categorical factors are coded as predictor variables.
For example:
[
Y=\beta_0+\beta_1X_1+\beta_2X_2+\varepsilon
]
Dummy or contrast variables represent the factor levels. Tests of sets of coefficients produce the corresponding ANOVA F tests.
This connection explains why:
- Contrast coding affects coefficient interpretation.
- Interactions can be represented as product terms.
- ANCOVA combines factors and continuous predictors.
- Unbalanced factorial analyses require clarity about tested hypotheses and sums of squares.
- Software may produce different tables when model order, coding or hypothesis definitions differ.
The official R documentation describes aov() as a wrapper around linear-model fitting and warns that interpretation can become more difficult in unbalanced designs (R Core Team, n.d.).
How ANOVA is used in modern research
ANOVA remains useful, but contemporary practice increasingly treats it as part of a wider modelling workflow rather than a single p-value-producing calculation.
A strong workflow includes:
- Define the experimental or observational unit.
- Specify factors, levels, outcomes and covariates before analysis.
- State the primary comparisons or contrasts.
- Plan sample size based on a meaningful effect and desired precision or power.
- Visualise raw data and model estimates.
- Fit a model appropriate to the outcome and dependence structure.
- Inspect residual and influence diagnostics.
- Report effect estimates, uncertainty and multiplicity adjustments.
- Conduct transparent sensitivity analyses.
- Preserve code, data definitions, software versions and decisions.
Unbalanced data
Unequal group sizes do not automatically make ANOVA invalid. However, in multifactor designs they can make main-effect and interaction hypotheses depend on contrast coding, model specification and the type of sums of squares used.
Researchers should not select Type I, Type II or Type III sums of squares merely because one option is a software default. They should define the scientific hypothesis being tested and confirm that the chosen table represents it.
Missing data
Classical repeated-measures ANOVA often requires complete records or imposes restrictive covariance assumptions. Deleting every participant with one missing occasion can reduce precision and may introduce bias.
Mixed-effects models can use incomplete repeated records under explicit assumptions, but they are not a universal cure. The reason for missingness and the model’s assumptions must still be examined.
Software for ANOVA
ANOVA is available in:
- R
- Python
- SPSS
- SAS
- Stata
- Minitab
- JMP
- JASP
- jamovi
- MATLAB
- Excel and spreadsheet add-ins
Basic one-way ANOVA in R
model <- aov(score ~ method, data = study_data)
summary(model)
TukeyHSD(model)
For unequal variances:
oneway.test(score ~ method, data = study_data, var.equal = FALSE)
The R documentation notes that aov() is most straightforward for balanced designs. More complex or unbalanced models require explicit consideration of contrasts, tested hypotheses and model structure.
Basic one-way ANOVA in Python
from scipy import stats
f_statistic, p_value = stats.f_oneway(
method_a,
method_b,
method_c
)
print(f_statistic, p_value)
For model-based factorial analysis, Python users commonly fit an ordinary least-squares model and generate an ANOVA table through a modelling package. The selected formula, contrast coding and sums-of-squares specification must match the intended hypothesis.
Software output should not replace reasoning
Before interpreting output, verify:
- The outcome and factor variables were entered correctly.
- Categorical variables were not accidentally treated as continuous numbers.
- Repeated observations have a subject identifier.
- Missing values were handled as intended.
- The correct error term was used.
- Interaction terms were included where required.
- The reported effect size is correctly named.
- Post-hoc p-values are adjusted.
- Degrees of freedom and corrections are understood.
- Model diagnostics support the analysis.
Using artificial intelligence with ANOVA
Generative AI can assist with:
- Drafting R or Python code.
- Explaining software output in simpler language.
- Creating simulated practice datasets.
- Translating formulas into code.
- Producing documentation or code comments.
- Identifying questions to discuss with a statistician.
AI should not be trusted to select or interpret an ANOVA without verification. It may overlook the experimental unit, dependence, missing data, factor coding, unequal variances, interaction structure or software-specific defaults.
AI verification checklist
Before using AI-generated statistical work:
- Confirm that the proposed model matches the research design.
- Check every variable’s measurement level.
- Verify the experimental unit and independence structure.
- Inspect the data directly.
- Run and understand the code rather than copying it blindly.
- Check package and software documentation.
- Compare the output with a second validated method when practical.
- Confirm all formulas, degrees of freedom and effect sizes.
- Preserve the prompts, code and analytical decisions if reproducibility requires them.
- Do not upload identifiable, confidential or restricted research data to an unapproved service.
Advantages of ANOVA
- It provides a unified test of several means.
- It avoids conducting many unadjusted two-sample tests.
- It can estimate multiple factors in one model.
- It can test interaction effects.
- It supports planned contrasts and adjusted comparisons.
- It connects naturally with regression and experimental design.
- It can incorporate blocking and covariates through extensions.
- Its table provides a transparent decomposition of variation.
Limitations of ANOVA
- A significant omnibus result does not identify the groups that differ.
- Classical inference depends on assumptions about errors and study design.
- Means may be poor summaries for highly skewed or heavy-tailed outcomes.
- Outliers can have substantial influence.
- P-values do not measure practical importance.
- Standard repeated-measures ANOVA can be restrictive.
- Unbalanced factorial designs require carefully defined hypotheses.
- Missing data can complicate interpretation.
- A non-significant result does not establish equivalence.
- Incorrectly identifying the experimental unit can produce misleadingly small standard errors and p-values.
Common ANOVA mistakes
Treating repeated observations as independent
Measurements from the same participant, school, field or specimen are usually correlated. Ordinary between-groups ANOVA is not appropriate unless the data are first reduced to genuinely independent units in a justified way.
Checking normality only on the pooled outcome
A mixture of groups with different means may look non-normal even when the residuals within groups are reasonable. Diagnostics should reflect the fitted model.
Using a normality test as an automatic decision rule
Normality tests do not replace residual plots, design knowledge or sensitivity analysis.
Running many unadjusted t-tests
Testing every pair separately at the same unadjusted alpha level increases the chance of at least one false positive.
Ignoring an interaction
When a meaningful interaction exists, averaging over the interacting factor may obscure or reverse the pattern of interest.
Reporting only the p-value
Readers also need descriptive statistics, mean differences, confidence intervals, effect sizes, sample sizes and methodological details.
Saying that the null hypothesis was accepted
A non-significant result normally means that the analysis did not provide sufficient evidence to reject the specified null hypothesis. It does not prove that all means are identical.
Interpreting non-significance as equivalence
Equivalence requires an equivalence margin and a suitable equivalence-testing procedure. It cannot be concluded from (p>.05) alone.
Choosing the analysis after viewing favourable results
Switching tests, contrasts, exclusion rules or transformations after seeing the outcome can inflate false-positive risk. Exploratory decisions should be labelled and reported transparently.
How to report ANOVA results in APA 7th style
A complete report normally includes:
- The type of ANOVA.
- Factors and levels.
- Whether observations were independent or repeated.
- Group sample sizes.
- Means and standard deviations or other suitable summaries.
- Diagnostic procedures and important deviations.
- F statistic and degrees of freedom.
- Exact p-value unless it is smaller than .001.
- Effect size and, where possible, its confidence interval.
- Corrections used for repeated measures.
- Follow-up contrasts and multiplicity adjustment.
- Confidence intervals for important mean differences.
- A substantive interpretation.
Reporting template
A [type of ANOVA] examined the effect of [factor] on [outcome]. The analysis indicated [evidence/no clear evidence] of a factor effect, F([df1], [df2]) = [value], p = [value], [effect-size symbol] = [value]. [Adjusted post-hoc test or planned contrast] showed that [group comparison and direction], mean difference = [value], 95% CI [lower, upper], adjusted p = [value].
Reporting the worked example
Examination scores differed by teaching method, F(2, 12) = 14.00, p < .001, (\omega^2=.63). Tukey-adjusted comparisons indicated that Method B (M = 81.00) produced higher scores than Method A (M = 75.00), mean difference = 6.00, 95% CI [2.92, 9.08], adjusted p < .001, and Method C (M = 77.00), mean difference = 4.00, 95% CI [0.92, 7.08], adjusted p = .012. Methods A and C did not differ clearly, adjusted p = .234.
The substantive conclusion should describe the estimated differences, not merely repeat that the result was “statistically significant.”
Conclusion
ANOVA is a flexible family of linear-model tests for examining mean differences across groups, factors and conditions. Its central F statistic compares variation associated with a model term against residual variation.
A reliable ANOVA requires more than obtaining a p-value. Researchers must choose a model that matches the design, examine assumptions, report effect sizes and uncertainty, and use appropriate contrasts or adjusted comparisons. When variances, outcome distributions or dependence structures do not fit classical ANOVA, Welch procedures, generalised models, robust methods or mixed-effects models may provide better answers.
