Experimental Design – Types, Methods, Guide

Experimental design is the structured plan used to test whether changes in one or more independent variables cause changes in an outcome. It specifies the treatments, experimental units, comparison conditions, assignment procedure, measurements, controls, sample size, and analysis strategy before data collection begins.

Experimental design determines how an experiment will answer a research question. A well-designed experiment does more than compare groups: it creates a defensible connection between the intervention or manipulated variable and the observed outcome.

This guide explains the principles, major types, planning process, examples, validity threats, statistical considerations, ethics, digital tools, and modern practices involved in experimental design.

Key Takeaways

• Experimental design is the advance plan for manipulating variables, assigning experimental units, measuring outcomes, and analyzing results.
• Random assignment supports causal inference, whereas random sampling supports population generalizability.
• A design may be described in several ways at once, such as a randomized, between-subjects, 2 × 2 factorial field experiment.
• The experimental unit—not simply the number of observations—determines the effective sample size.
• Sample-size, exclusion, outcome, and analysis decisions should be planned before researchers examine the final results.
• Strong experiments use appropriate controls, randomization, replication, blinding where feasible, and transparent reporting.

What Is Experimental Design?

Experimental design is the process of deciding how an experiment will be conducted so that the resulting evidence can answer a research question with minimal bias and uncertainty.

The design normally specifies:

• The research question and hypothesis.
• The independent variable or treatment.
• The dependent variable or outcome.
• The experimental units.
• The treatment and comparison conditions.
• How units will be assigned to conditions.
• Which variables will be controlled, blocked, or measured.
• When and how outcomes will be measured.
• The required sample size.
• The planned statistical analysis.
• The procedures for handling exclusions, missing data, and deviations.

In statistical design of experiments, often abbreviated as DOE, researchers deliberately vary one or more factors to observe their effects on one or more responses. The objective is to obtain valid and useful information efficiently.

Why Is Experimental Design Important?

Experimental design is important because causal conclusions depend on how the data were produced, not only on how they were analyzed.

A poorly designed experiment may produce biased or ambiguous findings even when sophisticated statistical software is used. Common problems include unequal groups, uncontrolled confounding, inappropriate measurement, too few experimental units, selective exclusions, order effects, and analyses that do not match the assignment procedure.

A strong design helps researchers:

• Estimate causal effects more credibly.
• Separate treatment effects from alternative explanations.
• Reduce systematic bias.
• Use participants, time, animals, materials, and funding efficiently.
• Estimate uncertainty appropriately.
• Anticipate practical and ethical problems.
• Create a reproducible protocol.
• Select an analysis that corresponds to the research question.
• Report the study transparently.

Experimental Design, Experimental Research, and Design of Experiments

These terms overlap but are not always used in exactly the same way.

Term	Meaning
Experimental design	The plan specifying how variables, treatments, units, measurements, controls, and analyses will be organized.
Experimental research	Research in which investigators deliberately introduce an intervention or manipulate a variable to study its effects.
Design of experiments (DOE)	A statistical tradition focused on arranging factor combinations efficiently and estimating their effects on response variables.
Randomized controlled trial	An experiment in which eligible units are assigned randomly to intervention or comparison conditions.
Quasi-experiment	A study that evaluates an intervention but lacks full random assignment or another defining feature of a randomized experiment.
Observational study	A study in which researchers measure exposures and outcomes without assigning the exposure or intervention.

Experimental design is broader than choosing a statistical test. It concerns how the study is constructed before the data exist.

Core Components of an Experiment

Research question

The research question identifies the causal relationship the experiment is intended to investigate.

A useful experimental question names:

• The target population.
• The intervention or exposure.
• The comparison condition.
• The outcome.
• The relevant setting or period.

Example: Does receiving weekly retrieval-practice quizzes, compared with rereading the same material, improve final examination scores among first-year university students?

Hypothesis

A hypothesis is a testable prediction about the expected relationship between variables.

Directional hypothesis: Students receiving retrieval-practice quizzes will achieve higher examination scores than students assigned to rereading.

Null hypothesis: Mean examination scores will not differ between the two conditions.

Independent variable

The independent variable is the factor manipulated or assigned by the researcher.

In the example above, the independent variable is the learning strategy:

• Retrieval practice.
• Rereading.

Each distinct value of an independent variable is called a level.

Dependent variable

The dependent variable is the outcome measured to evaluate the effect of the independent variable.

Possible dependent variables in the education example include:

• Examination score.
• Delayed retention score.
• Completion rate.
• Time spent studying.

Researchers should identify one primary outcome when the study is confirmatory. Measuring many outcomes and emphasizing only favorable results increases the risk of selective reporting.

Treatment condition

A treatment condition is a particular intervention or combination of factor levels applied to an experimental unit.

A study may have:

• One treatment and one control.
• Several treatment levels.
• Several active treatments.
• Multiple factor combinations.

Control or comparison condition

A control condition provides a benchmark for interpreting the treatment outcome.

Possible controls include:

• No treatment.
• Usual practice.
• A placebo.
• A wait-list condition.
• An attention control.
• An active alternative treatment.
• A baseline or pre-intervention measurement.

Not every experiment requires a no-treatment group. The appropriate comparator depends on the research question.

Experimental unit

The experimental unit is the smallest unit independently assigned to a treatment.

It may be:

• A person.
• A classroom.
• A school.
• A hospital.
• A village.
• An animal.
• A laboratory plate.
• A plant plot.
• A store.
• A website visitor.

Suppose four classrooms are assigned to two teaching methods and 100 students are tested. The students provide observations, but the classroom may be the experimental unit because the treatment was assigned at classroom level.

Counting all 100 students as though they had been independently assigned can overstate precision. This problem is often called pseudoreplication.

Extraneous and confounding variables

An extraneous variable is any variable other than the intended independent variable that may influence the outcome.

A confounding variable is an extraneous variable that varies systematically with the treatment and therefore offers an alternative explanation for the observed effect.

For example, if one teaching method is always delivered in the morning and the other in the evening, time of day is confounded with teaching method.

Principles of Good Experimental Design

The classical principles of experimental design are commonly summarized as control, randomization, and replication. Modern experimental research also emphasizes blocking, blinding, allocation concealment, prespecification, appropriate measurement, and transparent reporting.

1. Manipulation

The researcher deliberately changes at least one factor.

Without manipulation or assigned intervention, a study is normally observational rather than experimental.

2. Control

Control means preventing alternative explanations from varying systematically across conditions.

Researchers may control extraneous influences through:

• Standardized procedures.
• Comparison groups.
• Eligibility criteria.
• Environmental control.
• Random assignment.
• Blocking or stratification.
• Matching.
• Counterbalancing.
• Statistical adjustment.

Control does not mean making every participant or situation identical. It means creating a fair comparison relevant to the causal question.

3. Randomization

Randomization uses a chance process to assign experimental units to conditions or determine treatment order.

Its purposes include:

• Reducing deliberate or unconscious allocation bias.
• Distributing measured and unmeasured characteristics across conditions on average.
• Supporting probability-based statistical inference.
• Preventing treatment assignment from being determined by prognosis or researcher preference.

Randomization does not guarantee that every baseline characteristic will be perfectly equal in a particular sample.

4. Replication

Replication means applying treatments to multiple independent experimental units.

Replication allows researchers to estimate natural variability and distinguish systematic treatment effects from random variation.

Repeated measurements from the same experimental unit are not necessarily independent replications.

5. Blocking

Blocking groups similar experimental units before treatment assignment. Treatments are then randomized within each block.

Examples include blocking by:

• School.
• Age group.
• Baseline disease severity.
• Laboratory batch.
• Field location.
• Device type.
• Prior achievement level.

Blocking can improve precision when the blocking variable is strongly related to the outcome.

6. Blinding or masking

Blinding prevents particular people from knowing the assigned condition.

Depending on the study, blinding may involve:

• Participants.
• Intervention providers.
• Outcome assessors.
• Data managers.
• Laboratory technicians.
• Statistical analysts.

Blinding may not be possible in interventions such as psychotherapy, teaching methods, or surgery. Researchers should still blind outcome assessment or analysis where feasible and report what could not be blinded.

7. Allocation concealment

Allocation concealment prevents the person enrolling or assigning units from predicting the next treatment assignment.

It occurs before or during assignment and should not be confused with blinding, which concerns knowledge of assignments after allocation.

8. Prespecification

Prespecification means deciding important design and analysis choices before examining the final outcome data.

A protocol or preregistration may specify:

• Primary and secondary outcomes.
• Sample-size justification.
• Eligibility criteria.
• Exclusion rules.
• Randomization method.
• Statistical models.
• Covariates.
• Transformations.
• Missing-data procedures.
• Subgroup analyses.
• Stopping rules.

Prespecification does not prohibit exploratory analysis. It makes the distinction between planned and exploratory analysis transparent.

Main Ways to Classify Experimental Designs

There is no single complete list of experimental designs. The same experiment can be classified along several dimensions.

Classification question	Common categories
Was treatment assigned randomly?	Randomized, quasi-experimental, pre-experimental
Who receives each condition?	Between-subjects, within-subjects, matched-pairs
How many independent variables are manipulated?	Single-factor, factorial
What is randomized?	Individual, cluster, split-plot unit
When are outcomes measured?	Posttest-only, pretest–posttest, repeated longitudinal measurements
How are nuisance variables handled?	Completely randomized, blocked, stratified, Latin square
Where is the experiment conducted?	Laboratory, field, online, clinical, industrial
Can the design change during the study?	Fixed, group-sequential, adaptive
How are factor combinations selected?	Full factorial, fractional factorial, response-surface design

A precise study description may therefore be:

A cluster-randomized, two-arm, parallel-group field experiment with pretest and posttest measurements.

Types of Experimental Design

Completely randomized design

In a completely randomized design, all eligible experimental units are assigned to treatments using one randomization process.

Example: Sixty laboratory samples are assigned randomly to three temperature conditions.

Best used when: Experimental units are reasonably homogeneous and no strong nuisance variable requires blocking.

Main advantage: Simple to implement and analyze.

Main limitation: Chance imbalance may reduce precision, especially in small samples.

Randomized block design

In a randomized block design, units are grouped into blocks based on a relevant characteristic, and treatments are randomized within each block.

Example: Students are grouped by baseline achievement level, after which students within each group are assigned to one of two teaching methods.

Advantage: Can reduce unexplained variability and improve treatment comparisons.

Limitation: Blocking on an irrelevant variable may add complexity without improving precision.

Between-subjects design

In a between-subjects design, each participant or unit receives only one experimental condition.

Example: One group uses a new learning platform, while another group uses the existing platform.

Advantages:

• Avoids carryover and many order effects.
• Suitable for irreversible or long-lasting interventions.
• Often easier for participants to complete.

Limitations:

• Usually requires more participants.
• Individual differences contribute to error.
• Small groups may be imbalanced despite randomization.

Within-subjects design

In a within-subjects design, each participant or unit experiences every condition.

Example: Each participant completes a memory task once in silence and once with background noise.

Advantages:

• Each participant serves as their own control.
• Individual differences are reduced.
• Fewer participants may be needed for comparable precision.

Limitations:

• Practice, fatigue, learning, and carryover effects may occur.
• Participation may take longer.
• Some interventions have effects that cannot be reversed or washed out.

Repeated-measures design

A repeated-measures design obtains multiple measurements from the same experimental unit.

It may involve:

• The same outcome measured at several times.
• The same person receiving several conditions.
• Multiple trials nested within each participant.

Repeated measures are correlated and should not normally be analyzed as independent observations.

Crossover design

In a crossover design, participants receive two or more treatments in a planned sequence.

For example:

• Sequence 1: Treatment A, washout, Treatment B.
• Sequence 2: Treatment B, washout, Treatment A.

Crossover designs are most appropriate when:

• The condition is stable.
• Treatment effects are temporary.
• A sufficient washout period is possible.
• Carryover can be controlled or modeled.

They are inappropriate when an intervention permanently changes the participant or when the underlying condition changes rapidly.

Matched-pairs design

In a matched-pairs design, units are paired according to important baseline characteristics. Members of each pair are then assigned to different conditions.

Matching variables might include:

• Age.
• Baseline score.
• Disease severity.
• Prior experience.
• Organizational size.

Matching is useful when the sample is small and a strong prognostic variable is available. However, exact matching is difficult, and the matched structure should be considered in the analysis.

Factorial design

A factorial design manipulates two or more independent variables in one experiment.

A 2 × 2 factorial design contains:

• Two independent variables.
• Two levels of each variable.
• Four treatment combinations.

The number of conditions in a full factorial design is the product of the numbers of factor levels:

Number of conditions = levels of Factor A × levels of Factor B × …

Thus:

• A 2 × 2 design has 4 conditions.
• A 2 × 3 design has 6 conditions.
• A 2 × 2 × 3 design has 12 conditions.

Factorial designs estimate:

• The main effect of Factor A.
• The main effect of Factor B.
• The interaction between A and B.

An interaction effect occurs when the effect of one factor depends on the level of another factor.

Full factorial design

A full factorial design includes every possible combination of factor levels.

It provides complete information about the included main effects and interactions but may become expensive as the number of factors increases.

Fractional factorial design

A fractional factorial design uses a carefully selected subset of factor combinations.

It is commonly used to screen many factors efficiently. The trade-off is that some effects may be aliased or confounded with other effects.

Response-surface design

Response-surface methodology studies how quantitative factors influence an outcome and helps identify an optimal combination.

Common applications include:

• Manufacturing.
• Chemical processes.
• Agriculture.
• Formulation development.
• Engineering optimization.

Central composite and Box–Behnken designs are common response-surface structures.

Cluster-randomized design

In a cluster-randomized design, groups rather than individuals are assigned to treatments.

Possible clusters include:

• Schools.
• Clinics.
• Villages.
• Workplaces.
• Families.
• Online communities.

Cluster randomization may prevent contamination when individuals in the same setting would otherwise share the intervention. However, outcomes within clusters tend to be correlated. Sample-size calculations and analyses must account for this intracluster correlation.

Split-plot design

A split-plot design uses more than one level of randomization.

For example, irrigation may be randomized to large field plots, while fertilizer is randomized to smaller subplots within each field.

Split-plot structures also occur in industrial, laboratory, educational, and organizational research when some factors are difficult or expensive to change.

Posttest-only control-group design

Units are assigned to treatment and control conditions, and the outcome is measured only after the intervention.

This design avoids pretest sensitization and may be efficient when randomization is credible.

Pretest–posttest control-group design

The outcome is measured before and after treatment in both intervention and comparison groups.

This allows researchers to:

• Assess baseline levels.
• Estimate change.
• Improve precision through baseline adjustment.
• Examine whether attrition differs by condition.

A pretest–posttest structure is not automatically a true experiment. The presence or absence of random assignment remains important.

Solomon four-group design

The Solomon four-group design combines posttest-only and pretest–posttest groups:

1. Pretest, treatment, posttest.
2. Pretest, control, posttest.
3. Treatment, posttest only.
4. Control, posttest only.

It can assess whether taking the pretest changes participants’ responses to the intervention. Its main disadvantage is the larger sample and additional complexity required.

Adaptive design

An adaptive design allows prospectively planned modifications based on accumulating data while preserving the validity and integrity of the study.

Possible adaptations include:

• Stopping early for success or futility.
• Dropping ineffective treatment arms.
• Modifying allocation probabilities.
• Adjusting sample size.
• Selecting doses.
• Enriching the study population.

An unplanned change made after looking at favorable results is not a valid adaptive feature. Adaptation rules, decision thresholds, simulations, and error control should be defined before the relevant data are examined.

True, Quasi-Experimental, and Pre-Experimental Designs

True experimental design

A true experiment normally includes:

• Deliberate manipulation of an independent variable.
• A comparison condition.
• Random assignment to conditions.
• Controlled and standardized measurement.

Randomized experiments generally provide the strongest design-based protection against confounding, although poor implementation, attrition, noncompliance, measurement bias, or selective reporting can still weaken conclusions.

Quasi-experimental design

A quasi-experimental design evaluates an intervention but does not use full random assignment.

Common quasi-experimental structures include:

• Nonequivalent control-group design.
• Interrupted time-series design.
• Regression discontinuity design.
• Controlled before-and-after study.
• Difference-in-differences design.
• Natural experiment.

Quasi-experiments can provide valuable causal evidence when randomization is impractical or unethical, but they require stronger assumptions and careful assessment of alternative explanations.

Pre-experimental design

Pre-experimental studies usually offer limited control over threats to validity.

Examples include:

• One-shot case study.
• One-group pretest–posttest design.
• Static-group comparison without random assignment.

These designs may be useful for piloting feasibility or generating hypotheses, but treatment effects are difficult to separate from history, maturation, selection, regression to the mean, and other influences.

Between-Subjects vs. Within-Subjects Design

Feature	Between-subjects	Within-subjects
Condition exposure	Each participant receives one condition	Each participant receives all conditions
Participant requirements	Usually higher	Usually lower
Individual differences	Contribute to error	Largely controlled within each person
Order effects	Usually absent	Often possible
Carryover	Usually absent	May be serious
Dropout impact	Usually affects one condition	May create incomplete repeated records
Suitable for permanent interventions	Yes	Usually no
Common analysis	Independent-groups model	Paired or repeated-measures model

Choose a between-subjects design when:

• Exposure to one condition would influence another.
• The treatment is irreversible.
• The study involves long-term interventions.
• Participants cannot reasonably complete all conditions.

Choose a within-subjects design when:

• Participant differences are large.
• Treatment effects are temporary.
• Carryover can be avoided or controlled.
• Recruitment is difficult.
• Repeated testing is scientifically meaningful.

Counterbalancing in Within-Subjects Designs

Counterbalancing varies the order in which conditions are presented.

For two conditions, researchers may assign:

• Half of the participants to A then B.
• Half to B then A.

For several conditions, possible methods include:

• Complete counterbalancing.
• Random order.
• Latin square arrangements.
• Balanced Latin squares.

Counterbalancing distributes order effects across conditions, but it does not necessarily eliminate carryover. Washout periods, alternative stimuli, practice sessions, or a between-subjects design may still be needed.

How to Design an Experiment: 12 Steps

Step 1: Define the causal question

State exactly what causal effect the study is intended to estimate.

A strong question identifies the population, treatment, comparator, outcome, and relevant time frame.

Step 2: Review theory and prior evidence

Determine:

• What is already known.
• Which mechanisms are plausible.
• What outcomes have been validated.
• What effect sizes are scientifically meaningful.
• Which design problems occurred in earlier research.

Step 3: Specify the hypothesis and estimand

The estimand is the precise quantity the study aims to estimate.

Examples include:

• The average effect of assignment to treatment.
• The effect among participants who comply with treatment.
• The effect at a particular follow-up time.
• The difference between two doses.

Defining the estimand clarifies the design, outcome, population, and analysis.

Step 4: Define the experimental unit

Identify what will actually be assigned independently.

Do not confuse:

• Experimental units.
• Measurement units.
• Repeated observations.
• Technical replicates.

Step 5: Operationalize the variables

Specify how each construct will be manipulated or measured.

For each outcome, document:

• Instrument.
• Scale.
• Timing.
• Scoring method.
• Reliability or measurement properties.
• Minimum meaningful difference where relevant.

Step 6: Select the treatment and comparator

Describe:

• Intervention content.
• Dose or intensity.
• Duration.
• Delivery method.
• Provider training.
• Control condition.
• Procedures for monitoring adherence.

Step 7: Choose the design structure

Decide whether the study will be:

• Between-subjects or within-subjects.
• Individual- or cluster-randomized.
• Single-factor or factorial.
• Posttest-only or pretest–posttest.
• Completely randomized or blocked.
• Fixed or adaptive.
• Laboratory, field, online, or clinical.

Step 8: Identify and control nuisance variables

List factors that might affect the outcome.

Decide whether each variable should be:

• Held constant.
• Randomized.
• Blocked.
• Counterbalanced.
• Measured as a covariate.
• Addressed through eligibility criteria.
• Left to vary to improve generalizability.

Step 9: Plan assignment, concealment, and blinding

Specify:

• Sequence-generation method.
• Allocation ratio.
• Block sizes, if used.
• Stratification variables.
• Who generates the sequence.
• Who enrolls units.
• How assignment is concealed.
• Who will be blinded.

Step 10: Determine sample size

Plan sample size using:

• The primary outcome.
• Meaningful effect size.
• Expected variability.
• Type I error rate.
• Desired statistical power.
• Allocation ratio.
• Repeated-measure correlation.
• Cluster correlation where applicable.
• Anticipated attrition.
• Multiplicity adjustments where necessary.

Simulation may be preferable for complex, multilevel, adaptive, or nonlinear designs.

Step 11: Write the analysis plan

Specify:

• Primary statistical model.
• Unit of analysis.
• Planned covariates.
• Treatment contrasts.
• Interaction tests.
• Missing-data procedure.
• Outlier and exclusion rules.
• Assumption checks.
• Sensitivity analyses.
• Subgroup analyses.
• Multiplicity strategy.

Step 12: Pilot, preregister, and finalize the protocol

A pilot can test:

• Recruitment.
• Comprehension.
• Intervention delivery.
• Timing.
• Equipment.
• Data capture.
• Outcome variability.
• Potential floor or ceiling effects.

Pilot results should not be treated automatically as definitive treatment-effect evidence.

Before the confirmatory study, finalize the protocol, obtain ethical approval where required, and preregister the hypotheses and analysis plan when appropriate.

Worked Example: A 2 × 2 Factorial Education Experiment

Research question

Do retrieval-practice quizzes and personalized feedback improve delayed learning among university students, and does their combination produce an additional benefit?

Factors

Factor A: Retrieval practice

• A0: No weekly quiz.
• A1: Weekly retrieval-practice quiz.

Factor B: Feedback

• B0: Standard answer key.
• B1: Personalized explanatory feedback.

Conditions

A 2 × 2 factorial design produces four conditions:

1. No quiz + standard answer key.
2. No quiz + personalized feedback.
3. Weekly quiz + standard answer key.
4. Weekly quiz + personalized feedback.

Experimental unit

The experimental unit is the individual student if students can be assigned independently without substantial contamination.

If the intervention must be delivered to whole classes, the class becomes the experimental unit and a cluster-randomized design is required.

Primary outcome

Delayed examination score four weeks after the intervention.

Secondary outcomes

• Immediate test score.
• Study time.
• Course completion.
• Student satisfaction.

Assignment

Eligible students are randomly assigned in equal proportions to the four conditions. Randomization is stratified by baseline achievement level.

Main effects

The retrieval-practice main effect compares the average of Conditions 3 and 4 with the average of Conditions 1 and 2.

The feedback main effect compares the average of Conditions 2 and 4 with the average of Conditions 1 and 3.

Interaction

An interaction exists if the benefit of personalized feedback differs depending on whether students also receive retrieval-practice quizzes.

The interaction can be represented conceptually as:

Interaction = effect of feedback with quizzes − effect of feedback without quizzes

Analysis

A linear model could include:

• Retrieval-practice condition.
• Feedback condition.
• Retrieval × feedback interaction.
• Baseline achievement.
• Prespecified stratification variables.

If allocation occurs by class, a multilevel or cluster-aware model is needed.

How to Choose the Right Experimental Design

Research situation	Often appropriate
Treatment has a permanent effect	Between-subjects design
Participants can complete all short, reversible conditions	Within-subjects design
Two or more interventions may work together	Factorial design
Participants naturally occur in schools or clinics	Cluster-randomized design
A strong baseline characteristic predicts the outcome	Randomized block or stratified design
Two temporary treatments are compared in stable participants	Crossover design
Many industrial factors require efficient screening	Fractional factorial design
Optimal settings of continuous factors are needed	Response-surface design
Randomization is not feasible	Strong quasi-experimental design
Pretesting may alter responses	Posttest-only or Solomon four-group design
Planned changes may be needed during a trial	Prospectively specified adaptive design

The design should follow the scientific question. Researchers should not choose a design simply because a particular software package or statistical test is familiar.

Sample Size and Statistical Power

Statistical power is the probability that a study will detect a specified effect when that effect exists under the assumed model.

Sample-size planning normally requires:

• A clearly identified primary outcome.
• A minimum scientifically important effect.
• An estimate of outcome variability.
• A significance threshold.
• Desired power.
• Expected attrition.
• Design-specific adjustments.

Design features that affect sample size

Within-subject correlation: Strong correlation between repeated measurements may improve precision.

Clustering: Similarity among participants in the same cluster reduces the amount of independent information.

Unequal allocation: Assigning more units to one condition may reduce efficiency unless justified by cost, ethics, or other constraints.

Multiple comparisons: Testing many primary hypotheses may require error-rate adjustment.

Interactions: Detecting interactions often requires a larger sample than detecting main effects.

Noisy measurement: Unreliable outcomes increase variance and reduce power.

A small study may be ethical and useful when its purpose is feasibility. An underpowered confirmatory study, however, may waste resources and expose participants without providing a reliable answer. An unnecessarily large experiment can also waste resources and expose more participants or animals than needed.

Random Sampling vs. Random Assignment

Random sampling	Random assignment
Selects units from a target population	Allocates enrolled units to conditions
Supports representativeness	Supports comparable treatment groups
Primarily affects external validity	Primarily affects internal validity
Occurs before enrollment is complete	Occurs after eligibility and enrollment
Often difficult in laboratory research	Central to randomized experiments

A study can use convenience sampling followed by random assignment. Such a study may estimate a credible causal effect within the recruited sample while still having limited population generalizability.

Controlling Bias and Confounding

Standardization

Use the same instructions, timing, equipment, setting, and outcome procedures across conditions except for the intended manipulation.

Random assignment

Assign units using a genuinely random method rather than alternation, date of birth, participant preference, or researcher judgment.

Blocking and stratification

Use a small number of strongly prognostic baseline variables when imbalance would be especially damaging.

Blinded outcome assessment

When participants or intervention providers cannot be blinded, independent assessors may still be kept unaware of treatment assignment.

Objective outcomes

Automated or objective measurements can reduce some forms of observer bias, although measurement systems still require validation and quality control.

Manipulation checks

A manipulation check examines whether the intervention changed the intended intermediate state.

For example, a noise manipulation might be checked using measured decibel levels and participant perception ratings.

A manipulation check is not a substitute for the primary outcome and should not be used carelessly to exclude participants after seeing results.

Contamination control

Contamination occurs when control units receive elements of the intervention.

Possible responses include:

• Cluster randomization.
• Separate intervention sessions.
• Access controls.
• Monitoring co-interventions.
• Measuring contamination.
• Intention-to-treat analysis.

Threats to Validity

Internal validity

Internal validity concerns whether the observed outcome difference can credibly be attributed to the intervention.

Threat	Explanation	Possible response
Selection bias	Conditions differ before treatment	Randomization, matching, baseline assessment
History	External events affect outcomes	Concurrent controls, shorter study period
Maturation	Natural change occurs over time	Comparison group, multiple measurements
Testing effect	Pretesting changes later performance	Alternative forms, Solomon design
Instrumentation	Measurement changes across conditions or time	Calibration, standardized measurement
Regression to the mean	Extreme baseline values move toward average	Randomized comparison, repeated baselines
Attrition	Dropout differs by condition	Retention procedures, transparent flow, sensitivity analysis
Contamination	Control units receive treatment elements	Cluster allocation, separation, monitoring
Noncompliance	Assigned treatment is not received as planned	Adherence measurement, intention-to-treat analysis
Expectancy effects	Expectations influence behavior or assessment	Placebo or attention control, blinding
Researcher degrees of freedom	Decisions change after outcomes are known	Preregistration and transparent reporting

External validity

External validity concerns whether findings apply to other:

• Populations.
• Settings.
• versions of the intervention.
• Outcome measures.
• Historical periods.

Highly controlled laboratory experiments may have strong internal validity but limited realism. Field experiments may improve realism but introduce implementation variability.

Construct validity

Construct validity concerns whether the manipulation and measurement accurately represent the intended concepts.

A single short test may not adequately represent learning, depression, trust, productivity, or another complex construct.

Statistical-conclusion validity

Statistical-conclusion validity concerns whether the analysis supports an accurate conclusion about the magnitude and uncertainty of the effect.

Threats include:

• Low power.
• Invalid independence assumptions.
• Unreliable measurement.
• Selective analysis.
• Model misspecification.
• Uncorrected multiple testing.
• Inappropriate handling of missing data.
• Dichotomizing continuous variables unnecessarily.
• Reporting only p-values without effect estimates and uncertainty.

Matching the Analysis to the Design

The statistical analysis should represent the assignment and measurement structure.

Design feature	Analysis implication
Independent treatment groups	Independent-groups comparison or regression model
Repeated measurements	Paired, repeated-measures, or mixed-effects model
Blocking	Include block effects or use matched analysis
Factorial design	Estimate main effects and interactions
Cluster randomization	Account for clustering
Unequal assignment probability	Consider design weights or appropriate model specification
Baseline outcome measurement	Prespecified baseline-adjusted analysis is often efficient
Adaptive design	Use analysis preserving the planned operating characteristics
Multiple primary outcomes	Prespecify multiplicity control
Noncompliance	Distinguish assignment effects from treatment-received effects

The statistical test should not determine the design after data collection. Design and analysis should be developed together.

Ethical Considerations

Experiments involving humans, animals, organizations, or sensitive data may require formal ethical review.

Researchers should consider:

• Scientific and social value.
• Favorable balance of expected benefits and risks.
• Informed consent or an ethically justified alternative.
• Fair participant selection.
• Privacy and data protection.
• Additional safeguards for vulnerable participants.
• Compensation and undue influence.
• Adverse-event monitoring.
• Stopping criteria.
• Post-study responsibilities.
• Timely reporting of findings.
• Conflicts of interest.

For medical research involving human participants, researchers should consult the current 2024 Declaration of Helsinki and applicable national or institutional requirements rather than relying on an archived version.

Ethical approval does not correct weak methodology. A study that cannot answer its question may expose participants or use resources without sufficient scientific value.

Modern Experimental-Research Practices

Preregistration

Preregistration creates a time-stamped record of the study plan before data collection or analysis.

A useful preregistration identifies:

• Research questions.
• Hypotheses.
• Design.
• Primary outcomes.
• Sample-size plan.
• Exclusion criteria.
• Statistical models.
• Confirmatory and exploratory analyses.

Researchers may deviate from a preregistration when circumstances require it. Deviations should be dated, explained, and reported transparently.

Registered Reports

Registered Reports separate evaluation of the research question and method from evaluation of the findings.

In the typical process:

1. Researchers submit the introduction, method, and analysis plan before collecting or examining the final data.
2. The journal conducts peer review.
3. A methodologically sound proposal may receive in-principle acceptance.
4. Researchers complete the study and report the results regardless of whether they are statistically significant.

Reproducible workflows

Modern experiments increasingly use:

• Version-controlled protocols.
• Scripted data cleaning.
• Reproducible statistical code.
• Data dictionaries.
• Audit trails.
• Electronic laboratory notebooks.
• Electronic data-capture systems.
• Persistent identifiers for materials and protocols.
• Shared code and de-identified data where ethical and legal conditions permit.

Simulation-based design

Simulation allows researchers to evaluate how a planned design behaves under different assumptions.

It is especially useful for:

• Multilevel studies.
• Cluster-randomized trials.
• Adaptive designs.
• Nonlinear models.
• Missing data.
• Unequal allocation.
• Complex interactions.
• Small-sample corrections.

Online and platform experiments

Digital systems can assign users to different interfaces, messages, recommendation policies, or learning activities.

Online experiments should still address:

• Unit of randomization.
• Repeat users and device switching.
• Interference between users.
• Novelty effects.
• Peeking at results.
• Multiple metrics.
• Privacy.
• Consent or appropriate governance.
• Practical versus statistical significance.

Adaptive experiments

Adaptive designs can improve efficiency when modifications are planned in advance and their statistical consequences are evaluated.

They generally require more preparation, simulation, data monitoring, and governance than fixed designs.

Digital Tools for Experimental Design

Useful tool categories include:

Power and sample-size tools

Examples include:

• G*Power.
• R packages such as pwr, simr, and design-specific packages.
• Stata power commands.
• SAS procedures.
• PASS.
• Simulation scripts in R or Python.

Results should be checked against the exact design assumptions.

Randomization tools

Randomization can be implemented through:

• Validated trial-management systems.
• Statistical software.
• Reproducible R or Python code.
• Institutionally approved randomization services.

The random seed, algorithm, restrictions, and person responsible should be documented where appropriate.

Experimental-design software

DOE tools can generate and analyze:

• Full factorial designs.
• Fractional factorial designs.
• Screening designs.
• Mixture designs.
• Response-surface designs.
• Optimal designs.

Protocol and preregistration platforms

Examples include:

• Open Science Framework.
• Discipline-specific registries.
• Clinical trial registries.
• Institutional protocol systems.

Diagramming tools

A design diagram can clarify:

• Units.
• treatments.
• Assignment points.
• Repeated measurements.
• clustering.
• exclusions.
• analysis structure.

The NC3Rs Experimental Design Assistant is specifically intended to support animal-experiment planning, including randomization, blinding, and sample-size considerations. It does not replace specialist statistical advice.

Artificial Intelligence in Experimental Design

AI tools can assist with limited parts of experimental planning, such as:

• Generating an initial checklist.
• Suggesting possible confounders.
• Explaining unfamiliar terminology.
• Drafting neutral participant instructions.
• Creating example randomization or analysis code.
• Checking whether protocol sections are missing.
• Summarizing methodological guidance supplied by the researcher.
• Producing simulated datasets for code testing.

AI should not be treated as an independent methodological authority.

Researchers should:

1. Verify every design and statistical recommendation.
2. Test generated code with known examples and simulations.
3. Consult a statistician for complex or high-stakes designs.
4. Avoid entering confidential, identifiable, proprietary, or unpublished data into unapproved public systems.
5. Check institutional, funder, registry, and journal policies.
6. Disclose material AI use when required.
7. Preserve a record of important AI-assisted decisions.
8. Confirm all citations against the original source.

An AI-generated sample-size calculation is not trustworthy unless its formula, inputs, assumptions, and output have been independently checked.

Advantages of Experimental Design

Well-designed experiments can:

• Provide strong evidence about causal effects.
• Control treatment assignment.
• Reduce confounding through randomization.
• Test mechanisms and interactions.
• Compare several interventions efficiently.
• Produce replicable procedures.
• Support precise estimates when blocking or repeated measures are used.
• Inform decisions in medicine, education, policy, business, engineering, and science.

Limitations of Experimental Design

Experiments may be limited by:

• Ethical restrictions.
• Cost and time.
• Recruitment difficulties.
• Artificial settings.
• Noncompliance.
• Attrition.
• Contamination.
• Carryover effects.
• Measurement reactivity.
• Limited generalizability.
• Inability to manipulate some exposures.
• Complex implementation.
• Dependence on assumptions and protocol adherence.

Randomization strengthens causal inference but does not guarantee correct measurement, complete follow-up, representative recruitment, proper analysis, or honest reporting.

Common Experimental-Design Mistakes

Mistake 1: Starting data collection without a written protocol

Important choices then become vulnerable to changing expectations or observed results.

Mistake 2: Treating technical replicates as independent units

Repeated readings from one sample improve measurement precision but do not create additional independently assigned experimental units.

Mistake 3: Confusing random sampling with random assignment

A representative sample does not automatically eliminate confounding, and random assignment does not automatically make a sample representative.

Mistake 4: Choosing the statistical test after viewing the results

This can inflate error rates and encourage selective analysis.

Mistake 5: Using an arbitrary sample size

Statements such as “30 participants are always sufficient” are not defensible. Required sample size depends on the outcome, effect, variability, design, analysis, and error rates.

Mistake 6: Ignoring order and carryover effects

Within-subjects and crossover designs require explicit consideration of sequence, washout, practice, fatigue, and learning.

Mistake 7: Randomizing clusters but analyzing individuals as independent

This generally understates uncertainty.

Mistake 8: Measuring many outcomes without identifying a primary outcome

This increases ambiguity and the risk of selective emphasis.

Mistake 9: Excluding participants based on post-treatment information

Post-randomization exclusions can destroy the comparability created by randomization.

Mistake 10: Assuming a nonsignificant result proves no effect

A nonsignificant result may reflect limited precision. Researchers should examine the effect estimate and confidence interval.

Mistake 11: Reporting only p-values

Effect sizes, uncertainty intervals, raw descriptive information, and practical relevance are also necessary.

Mistake 12: Calling every pretest–posttest study an experiment

Without an assigned intervention and credible comparison strategy, a before-and-after study may provide weak causal evidence.

Experimental-Design Template

Researchers can adapt the following planning template.

Study objective

• What causal question will the experiment answer?
• Why is the question important?

Population and setting

• Who or what is eligible?
• Where will the experiment occur?
• What recruitment or sampling method will be used?

Experimental unit

• What is independently assigned?
• Are observations nested within larger units?

Intervention and comparator

• What does each condition receive?
• How long does the intervention last?
• How will fidelity and adherence be measured?

Variables

• Independent variables and levels.
• Primary dependent variable.
• Secondary outcomes.
• Potential moderators and mediators.
• Important nuisance variables.

Assignment

• Randomization method.
• Allocation ratio.
• Blocking or stratification.
• Allocation concealment.
• Counterbalancing or sequence generation.

Blinding

• Who can be blinded?
• How will blinding be maintained?
• How will unblinding be documented?

Measurement schedule

• Baseline measurements.
• Post-intervention measurements.
• Follow-up measurements.
• Instrument and scoring procedures.

Sample size

• Meaningful effect.
• Variability assumption.
• Alpha level.
• Target power.
• Attrition adjustment.
• Clustering or repeated-measures adjustment.

Analysis

• Primary model.
• Unit of analysis.
• Prespecified contrasts.
• Covariates.
• Interaction tests.
• Missing-data procedure.
• Multiplicity strategy.
• Sensitivity analyses.

Data quality

• Training.
• Calibration.
• Range checks.
• Duplicate or automated checks.
• Data dictionary.
• Audit trail.

Ethics

• Review and approval.
• Consent.
• Risk controls.
• Privacy.
• Adverse-event procedures.
• Dissemination plan.

Transparency

• Protocol location.
• Preregistration or registry.
• Code-sharing plan.
• Data-sharing plan.
• Reporting guideline.

How to Describe Experimental Design in a Research Paper

The methods section should allow a knowledgeable reader to understand and, where practical, reproduce the experiment.

A concise design statement might read:

We conducted a preregistered, randomized, two-group, parallel, between-subjects field experiment. Eligible students were allocated in a 1:1 ratio to retrieval-practice or rereading conditions using computer-generated block randomization stratified by baseline achievement. The primary outcome was delayed examination score at four weeks. Outcome assessors and the statistician were blinded to coded group assignments.

The full methods section should also explain:

• Participants and eligibility.
• Setting and recruitment.
• Experimental units.
• Intervention and comparator.
• Assignment and concealment.
• Blinding.
• Measures.
• Sample-size calculation.
• Exclusions.
• Statistical analysis.
• Ethics.
• Registration and protocol.
• Deviations from the original plan.

Randomized trial protocols should consult SPIRIT 2025, while reports of completed randomized trials should consult CONSORT 2025. Field-specific guidelines may add further requirements.

Conclusion

Experimental design is the advance structure that connects a causal question to the data needed to answer it. A credible design defines the experimental unit, treatments, comparator, assignment method, measurements, sample size, bias controls, and analysis before results are known.

The most appropriate design is not always the most complicated one. It is the design that answers the research question ethically, controls the most important alternative explanations, reflects the real data structure, and produces results that can be analyzed and reported transparently.

References

• Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Houghton Mifflin.
• Center for Open Science. (n.d.). Preregistration.
• Chan, A.-W., Boutron, I., Hopewell, S., et al. (2025). SPIRIT 2025 statement: Updated guideline for protocols of randomized trials. JAMA. https://doi.org/10.1001/jama.2025.4486
• Fisher, R. A. (1935). The design of experiments. Oliver & Boyd.
• Hopewell, S., Chan, A.-W., Collins, G. S., et al. (2025). CONSORT 2025 statement: Updated guideline for reporting randomized trials. JAMA, 333(22), 1998–2005. https://doi.org/10.1001/jama.2025.4347
• National Institute of Standards and Technology. (n.d.). NIST/SEMATECH e-Handbook of statistical methods: What is experimental design?
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
• U.S. Food and Drug Administration. (2019). Adaptive design clinical trials for drugs and biologics: Guidance for industry.
• World Medical Association. (2024). Declaration of Helsinki: Ethical principles for medical research involving human participants.