Skip to content

Recording an experiment

How to register a test in the Experiment Registry, record what each arm measured, and read the outcome the evidence gate decides — including why some results say nothing at all.

Updated

The Experiment Registry is where you write down what you tried, what you measured and what may honestly be concluded from it. It holds a hypothesis, the arms you compared, who saw them, the platform, the period, the metric you judged them on, the sample behind each arm, and then the outcome, the confidence band, your recommendation and whether the result carries beyond the workspace it was measured in.

Social Studio does not run the test for you. It does not split an audience, hold anything constant or serve one arm to one group and another arm to another. You run the test, and this records it — which is exactly why the registry is careful about what it will let you say afterwards.

The outcome is not yours to type. You supply the numbers; the same evidence gate the rest of Social Studio uses decides whether those numbers amount to anything, and it refuses to call a result proven unless a challenger genuinely beat the control.

BEFORE YOU START

  • The "view" permission to read experiments, "create" or "edit" to design and start one, and "administer" to record a result.
  • Access to the workspace the experiment belongs to.
  • At least two arms, one of which is the arm you already use. That one is the control.
  • At least five measured units in each of two arms before a comparison can be made at all. Below that the registry records no result rather than a weak one.
  • Recording an experiment costs no credits.

STEPS

  1. Open Workspaces, choose the workspace, and follow "Experiments" — or go to /app/workspaces/{workspace}/experiments.
  2. Fill in the design panel: a name, the hypothesis you are testing, the audience that was exposed, the platform, the date the period starts, and the metric you will judge the arms on.
  3. Name the two arms. Mark the thing you already do as the control and the thing you want to try as the challenger.
  4. Choose "Record as a draft". Nothing is claimed at this point; a draft is a design and can still be edited.
  5. When the test is genuinely under way, choose "Start". From then on the design is frozen, because changing a hypothesis or an arm after measurement has begun changes what the recorded numbers are evidence of.
  6. When the period ends, choose "Record the result" on the running experiment.
  7. For each arm, enter how many units were measured and the metric value per unit. Enter a rate, never a total: a total compares the size of the arms rather than how they did.
  8. Choose a recommendation — no recommendation, retest, discard, or adopt the winning arm — and say whether the result generalises.
  9. Submit. The outcome, the confidence band and any winner come back from the server.

WHAT YOU SHOULD SEE

The experiment listed with one of three outcomes. "Not enough evidence" means fewer than two arms reached the sample gate, so no comparison could be made at all. "Measured, not better" means the comparison was made and no challenger beat the control, including the case where the top two arms tied. "Proven" means a challenger cleared the gate, beat every other arm and beat the pooled average of the comparison it won.

A proven result also carries a confidence band reading "provisional" or "established". That band is a disclosed sample band and nothing more: "established" means at least three times the minimum sample, and no significance test is performed or implied. Every other outcome shows no band at all rather than a hedged one.

WHAT THIS WILL NOT DO

  • It will not split your audience, assign anybody to an arm, or serve different content to different people. The test happens outside Social Studio.
  • It will not let you record your own outcome, confidence or winner. Those fields are derived from the arms and a request that sets them is ignored.
  • It will not let you adopt a variant on an unproven result, or say a result carries beyond this workspace unless a challenger genuinely won. It refuses and says which claim it refused.
  • It will not verify that your arms were otherwise alike. Nothing recorded here can establish that, so every outcome is shown with the limitation that it is what the arms measured and not a proven cause.
  • It will not change what publishes. Adopting an arm records a decision; changing what actually goes out is done on the surfaces that already govern publishing.
  • It will not break a tie. Two arms at the same rate produce no winner.

WORTH KNOWING

The gate of five measured units per arm is the same threshold the performance recommendations and the proven-format library use. There is one answer in this product to "how much is enough", not four, and raising it in one place would raise it everywhere.

The registry deliberately keeps "not enough evidence" and "measured, not better" apart. The first says the comparison could not be made; the second says it was made and the challenger lost. Collapsing them would let an under-powered test read as a settled negative, and people abandon good ideas on the strength of that.

If you complete an experiment and an endpoint you registered is subscribed to it, an "experiment.completed" delivery is sent carrying the outcome, the band and the sample. It never carries a headline sentence, so a receiver cannot act on a summary that reads stronger than the result.

IF IT DOES NOT WORK

  • "Exactly one arm must be marked as the control." means you marked none or two. Two controls are two baselines, and a result measured against either of them is a different result.
  • "An experiment needs at least two arms." means a single arm was submitted. One arm has nothing to be compared against.
  • "The design of a running experiment cannot be edited." means the experiment has already started. Abandon it and design a new one rather than rewriting what is being measured.
  • A refusal naming the outcome, such as "This experiment recorded not_enough_evidence, so a variant cannot be adopted on the strength of it", is the causal-claim rule. Record "retest" or "discard", or run the test again with more units in each arm.
  • "administer permission required." means your role can read and design experiments but not turn one into a recorded result.
  • "You do not have access to this workspace." means your access does not reach it.

COMMON QUESTIONS

Does Social Studio decide who sees which arm?

No. It records a test you ran; it has no audience-splitting or serving mechanism of any kind.

Is the confidence a p-value?

No. It is a sample band with two values, and the wording that says so travels with every result. Nothing here performs a significance test.

Why did my clear-looking win come back as not enough evidence?

Because fewer than two arms reached five measured units. A large gap between two tiny samples is not evidence of anything, and the registry will not dress it up as one.

Can I edit a completed experiment?

No. Completed and abandoned are both final. What was recorded is what the product will cite, so it does not move.

  • What your recorded results support

    Reading the performance recommendations, the sample size every claim must clear, and the list of questions the recorded data cannot answer.

  • Reading your reporting

    What the workspace report totals, why a figure can say "Not reported" instead of zero, and the two ways a figure reaches it.

  • Receiving webhooks

    Registering an endpoint, the two events Social Studio sends, how to verify a signature, and the retry and replay behaviour you should build for.