Experiments and causation
Define the comparison before calling a change an experiment.
A later improvement does not prove which edit caused it. Define one treatment, a useful comparison and the evidence needed for a decision before changing the page.
Write one hypothesis and identify the unit.
State the reader task, the change and the observable difference you expect. Keep the unit clear: a page, a stable group of pages, a session or a participant is not interchangeable. Decide who receives a treatment and how assignment occurs before collecting results.
Separate the experimental question from a needed factual repair. Never keep a false claim, inaccessible component or broken task in a control merely to preserve a comparison. Fix unacceptable defects first. A test can compare truthful content orders, but the evidence, eligibility conditions, destinations and material qualifications should remain protected.
Keep a baseline with the same measurement definition.
Save the accepted content order, page set, versions, measurement configuration and observation window. Define the numerator, denominator, exclusions and unit of analysis for the chosen measure. For example, link events are not necessarily unique readers, and a click alone does not prove that a task was completed.
Confirm that collection is permitted and functioning before interpreting a baseline. Missing events, changed consent coverage or an altered event definition can make periods incomparable. Do not fill an unknown baseline with zero. Search Console report movements are useful contextual observations; they do not randomly assign visitors or establish the cause of a change.
Worked design: move a comparison before its background.
This fictional test concerns an established explanation that contains an answer, a comparison table, supporting background and a next-step link. The hypothesis is that putting the comparison immediately after the answer helps the intended readers find the condition that determines which route to choose. The only treatment is the order of those existing blocks.
Version A keeps answer, background, comparison and next-step link. Version B keeps answer, comparison, background and the same next-step link. The wording, data, destinations, metadata, access and qualifications stay fixed. Check that each version remains readable and truthful on supported narrow, keyboard and assistive surfaces.
For a controlled usability study, assign eligible participants to a version and give the same task: identify the applicable condition and choose the matching route. Record correct task completion and relevant errors, not simply scrolling. This can inform page usability; it is not an organic-ranking experiment. For a live page-family SEO test, define a separate comparable control/treatment design and search outcome plan; do not treat a visitor A/B result as proof of a crawler response.
This is a populated design proposal, not an executed test or a ready-to-run release. No participant has been exposed in this example. The unresolved sample, window and analysis choices keep it on hold; they are not filled with invented counts or success thresholds.
| Design field | Populated teaching plan |
|---|---|
| Eligible units | Recruited participants fitting the stated reader task, each assigned to A or B; eligibility and permitted exclusions fixed before recruitment |
| Treatment | Comparison moves before background; content and destination facts unchanged |
| Comparison | Assigned version A versus B in a controlled task study; live SEO test requires its own design |
| Task measure | Participants who correctly identify both condition and route / all assigned eligible participants, reported separately by A and B; record non-completion and missing observations separately, without silently dropping them |
| Harm checks | No hidden qualification, broken table, inaccessible route or inaccurate claim |
| Window and evidence | HOLD: sample, recruitment window and analysis rule remain unresolved. The research owner must set them and document exclusions/missing-data handling before any exposure |
| Decision owner | Research/editorial owner interprets evidence; release owner controls deployment and rollback |
Keep conditions comparable and record what cannot be controlled.
Use equivalent eligibility, tasks, devices and measurement definitions. Random assignment can reduce some selection differences when feasible; matching pages or comparing periods requires stronger discussion of remaining differences. Preserve assignment and exposure records appropriate to the approved purpose.
A content-order change can alter context even with the same words. Check that qualifications stay beside the claims they limit and that source/focus order remains meaningful. Avoid simultaneously changing title, anchor, destination, page speed, navigation and promotional activity if the question is about order. If another necessary change occurs, record it and reconsider the test rather than hide it.
List the reasons attribution could fail.
Potential confounders include seasonality, query or audience mix, search updates, competing releases, indexing changes, promotions, device differences, repeat visits, uneven assignment and instrumentation changes. A comparison group can also be affected by the same template change or by links from the treatment group.
In the teaching design, users familiar with version A might find its order faster through prior learning. A new campaign might bring beginners only to B. A tracking fault might undercount one version. A live search result may reflect a title change elsewhere, not the comparison order. Record which threats were checked and which remain plausible.
Plan uncertainty before reading the outcome.
Choose the decision threshold and analysis with someone competent in the design. Report the observation window, sample, missingness and uncertainty, including the possibility of no useful difference. Small, biased or highly variable evidence may support only a follow-up question. Repeatedly inspecting results until one looks favorable can mislead.
Do not manufacture a confidence interval, statistical significance or percentage lift without the underlying valid analysis. Distinguish an observed difference from a causal interpretation and from a general claim about all pages. A result in one audience or template can fail elsewhere even when the local study was sound.
Protect a rollback route and stop conditions.
Keep the prior accepted content, configuration, assignment logic where relevant and deployment reference. Stop for false claims, broken user tasks, loss of access, unacceptable privacy risk or invalid measurement. Name who can stop the test and how the accurate baseline is restored. Do not keep exposure running just to reach a planned date after a protected condition fails.
After rollback, inspect the answer, qualifications, table, links, metadata and intended visitor state again. Restoring a file is not proof that caches or shared templates returned to the accepted version. Keep the reason for stopping in the result record; a stopped test is not a successful treatment result.
Retest deliberately and report within scope.
A credible result can justify replication on another eligible cohort or time window with the same definitions. A contradictory result should reopen assumptions about audience, context and measurement instead of being discarded. Changes to the treatment create a new test question and need a new record.
The final handoff should include hypothesis, baseline, treatment, comparison, evidence, confounders, uncertainty, harm checks, stop/rollback events and next action. State “no attributable result” when that is what the design supports. This model does not report an experiment conducted by Semantec or promise search or commercial lift.
Sources and related learning
Google Search Console performance documentation.
The worked scenarios are fictional teaching records. They do not report live tests, client work or measured search outcomes.
Continue with semantic SEO learning and behavioral measurement and interpretation. Plan responsible behavioral evidence. Choose the next repair with evidence.