Experiment Log: What to Record Before, During and After

An experiment log is a dated record of what you tested, why you expected it to work, what you did, and what happened. A good one has a second job that most templates skip: it captures your prediction before the result arrives, so you can’t quietly rewrite it afterward. If you run product, growth or pricing experiments, that second job is where most of the learning lives.

This guide covers the fields that matter, a template you can copy, the habits that keep a log honest, and how a team log differs from a lab notebook.

TL;DR

• An experiment log should record the hypothesis, a probability, the metric and the kill criteria before the test runs, and the outcome after.
• Lab-grade logging rules apply to business tests too: date every entry and correct by adding, never by deleting.
• Keep the template short. Consistency over 15 to 20 entries beats a perfect form that gets abandoned.
• Review closed experiments to separate a bad decision from bad luck. Otherwise a lucky win teaches you the wrong lesson.
• A shared tool beats a scattered doc once more than a few people run tests at the same time.

What an experiment log is for

Scientists keep logs so that someone else can reproduce the work. The National Institutes of Health’s guidance for lab notebooks describes a record that covers formulating the question or hypothesis, designing the study, developing protocols, and generating and interpreting data. That is a useful standard to borrow even if your lab is a pricing page.

Business teams have a different problem. Rarely does anyone need to reproduce your onboarding test in a physical sense. The risk is that three months later nobody remembers what you expected, why you chose that metric, or what result would have made you stop. The log is the only witness.

A log serves three readers:

1. You, next week, when you need to remember exactly what changed in the test.
2. Your teammate, next quarter, who wants to know whether this idea was already tried.
3. Your future self, when the result is disappointing and your memory starts offering flattering explanations.

The third reader is the hardest to serve, and it shapes everything below.

The trap a plain log doesn’t fix

Most experiment log templates are built around results: what happened, which numbers moved, what we concluded. They’re fine for documentation. They do little for judgment.

Annie Duke calls the core problem “resulting”: judging a decision only by its outcome. Her point is that outcomes depend on both decision quality and luck, so a good decision can lead to a bad outcome and the reverse. An A/B test that wins on a two-week traffic spike may reflect nothing about the quality of your hypothesis. A test that fails because a competitor launched the same day may have been well designed.

If the log only records results, you will treat the winners as proof of skill and the losers as noise. That is the self-serving pattern the log was supposed to prevent.

The fix is simple to state and hard to do: write the reasoning down first. Calibration is trainable, but it needs a feedback loop where predictions are made before outcomes are known. Without that, hindsight bias contaminates the data.

The fields that earn their place

Here is a log structure that works for product, growth and strategy experiments. The first group is filled in before launch, the second during the run, the third at close.

Before the test: commit in writing

The confidence field deserves a note. “I think so” can’t be scored. “65%” can. Over many entries you can check whether your 70% calls land about 70% of the time. A Turbine analysis of calibration research reports that a roughly 45-minute training module measurably improved calibration scores across a study of 2,246 participants, which suggests the skill is trainable with structured practice.

During the run: log changes, don’t edit history

Experiments drift. Someone changes the audience, a bug appears, a holiday distorts traffic. Record each of these as a new dated line.

This follows a rule from research practice: the NIH guidance stresses that corrections should be tracked and dated rather than deleted, so the audit trail stays clean. For a business team the stakes are lower, but the logic is identical. If you delete the inconvenient note about the tracking bug, you lose the one fact that might explain the result.

Useful run-time fields:

• Environment and parameters: traffic split, audience, device mix, price points, tool versions
• Changes and incidents: what happened, when, and whether it touches the metric
• Interim checks: the date and what you saw, noting whether you were tempted to stop early

After the test: the verdict and the post-mortem

The “actual outcome” field is worth guarding. One public template describes it as a place for only actual facts, state and data. Keep interpretation for the verdict and post-mortem fields, so the two never blur.

The verdict “inconclusive” is underrated. Many teams force a win or a loss because ambiguity feels like failure. An honest inconclusive result, with a note on what sample size would have settled it, is more useful than a manufactured conclusion.

A short template you can copy today

If the table above looks heavy, start with this. Practitioners often recommend a four-line structure for decision journals: the decision, the reasoning, the expected outcome and a confidence rating. Adapt it for experiments by adding two lines:

Experiment: [name]            Date: [today]   Owner: [name]
1. What we are changing:
2. Why we expect it to work:
3. Expected result (number or range):
4. Confidence (0-100%):
5. Metric and win threshold:
6. What would prove us wrong:
--- after the run ---
7. What happened (facts only):
8. Verdict (won / killed / inconclusive) and the learning:

The same source notes that complex templates tend to get abandoned, and that you need something like 15 to 20 entries before patterns begin to show. Treat the first month as practice. Keep entries short enough that writing one takes five minutes.

If you want a fuller version, our decision log template is built to be copied in about ten minutes, and the seven-field decision log is a good step up once the short form feels easy.

Write the intent before you start testing

The habit of stating intent first is not unique to experiment logs. In software quality work, an exploratory testing charter fixes the mission and scope before a session begins. This exploratory testing charter guide shows how a written charter keeps a session focused. Your hypothesis, metric and kill criteria do the same job for a business experiment: they define what you are trying to learn before the data starts persuading you.

Habits that keep a log honest

A template doesn’t make a log trustworthy. These habits do.

Timestamp at the moment of writing. A log filled in on Friday for a test that started Monday is a memory, not a record. Use a tool or document that records when each entry was created.

Put the probability down, even when it feels artificial. Your first numbers will be rough. That’s expected. The point is the comparison over time.

Ask “wanna bet?” before you commit. If a colleague offered you even odds against your hypothesis, would you still take your side? If you hesitate, your stated confidence is probably too high. Lower it and write down why.

Do the pre-mortem while the test is still cheap to change. Imagine the test has failed and list the reasons. Teams often discover a flawed metric or a confounded audience at this stage, which costs a conversation instead of a quarter.

Review on a schedule. Read closed entries monthly. A log nobody reads is a diary.

Invite dissent in the review. Ask someone who didn’t run the test to challenge the verdict. The goal is a team that rewards finding the truth over defending a position.

Lab notebook vs. team experiment log

The two share a skeleton but differ in what they protect.

Borrow the rigor from the left column and the judgment fields from the right. You get a log that is both traceable and educational.

Where spreadsheets and docs stop working

A spreadsheet or shared doc is a reasonable place to start, and for one person running a few tests it can be enough. Trouble appears when:

• several people run tests at once and nobody can see overlaps
• entries get edited after the fact, with no trace
• confidence figures live in a column nobody ever aggregates
• post-mortems get skipped because there’s no stage that demands them
• old experiments are impossible to find when a similar idea resurfaces

At that point the log needs stages, ownership and a way to compare predictions with results across dozens of entries. That’s the gap a dedicated tool fills.

Our perspective: an experiment is a bet

We built Betlog on one idea from Annie Duke’s Thinking in Bets: every decision is a bet, a commitment made with incomplete information, where the outcome depends on decision quality and luck. An experiment is the cleanest example. You spend time and money to buy information, and you rarely know the odds.

So we don’t think of an experiment log as a results archive. We think of it as a receipt. It shows what you believed, how strongly, what you were willing to trade away, and what would have changed your mind. Each bet in Betlog captures the hypothesis, the expected outcome, the confidence as a probability, the trade-offs accepted, the deciding metrics and what would prove it wrong. It then moves through Idea, Prioritized, Running, Reviewing and Decided, and closes as Won, Killed or Inconclusive with a post-mortem that separates skill from luck.

The reason we insist on writing first is that memory is a poor witness. After a result, everyone remembers having expected it. A timestamped record removes that option. Over time it also shows something a single test can’t: whether your 70%-confident experiments actually win about seven times in ten, and where your team runs hot or cold.

If your experiments live in a doc that anyone can edit, ask yourself one question. Could you prove, today, what you predicted for the last test you ran? If not, that is the thing to fix first.

— Cesar

Run your experiment log in Betlog

If you lead a startup or a small team and want these fields, stages and post-mortems in one shared place, Betlog is built for that. It’s one plan: $39 per month or $390 per year. You can see what’s included on the pricing page, or start from the Betlog home page.

Want to go deeper on the habits first? Our guide on checking whether your 70% bets hit 70% walks through the calibration review, and the nine-field decision audit shows how to stop rewarding luck.

FAQ

What should an experiment log include?

At minimum: a hypothesis with its reasoning, an expected result, a confidence probability, the metric and win threshold, and what would prove you wrong. After the test, add the actual outcome, a verdict, and one learning. Professional research logs also record conditions and parameters so a third party could reproduce the work.

How is an experiment log different from a decision log?

An experiment log is tied to a test with a defined run and metric. A decision log covers any choice, including ones with no test. The best practice is the same for both: write the reasoning and your confidence before the outcome is known, and review later. Many teams keep experiments as a special kind of decision in one system.

Can I use a spreadsheet as an experiment log?

Yes, to start. Add a created-at date to every row, never overwrite old entries, and include a probability column you can score later. Once several people run tests and reviews begin to slip, a tool with stages and permanent records usually saves effort.

How many entries do I need before it helps?

A single entry helps you think more clearly that day. Patterns in your judgment take longer: one practitioner guide suggests 15 to 20 entries before they start to show. Keep each entry short so you reach that number.

Sources

• Keeping Lab Notebooks in the NIH IRP: Basic Principles and Best Practices
• Annie Duke: a good framework for two kinds of uncertainty
• Decision Journal: How to Audit Your Own Thinking
• Turbine: calibration research
• The Decision Journal, Nudges at Work
• Experiment Log, public template page

Recommended

• Template First Lessons Learned Register That PMs Actually Use
• Decision Log Template for Project Managers You Can Copy in 10 Minutes
• Check If Your 70% Bets Hit 70%: Decision Journal for Founders
• Preserve Institutional Memory: Log the ‘Why’ in 2 Minutes for Founders