Measurement and testing

7.1 Use a decision metric tree

Do not ask one metric to explain the whole flow.

Delivery health

• Accepted, bounced, blocked, complained, and unsubscribed.
• Watch for abrupt changes after sender, audience, or infrastructure changes.

Message engagement

• Clicks on the intended action.
• Replies when replies are a real part of the design.
• Resource access where privacy and tooling allow.
• Treat opens as directional because client privacy features can distort them.

Journey progression

• Movement from resource to product page.
• Checkout starts or purchases when reliably attributed.
• Customer exits and branch accuracy.

Quality signals

• Support questions that reveal unclear copy.
• Replies that reveal audience language.
• Refund reasons or mismatch reports.
• QA defects and broken-link incidents.

7.2 Define metrics before launch

For each metric, record:

• Definition.
• Data source.
• Known limitation.
• Review frequency.
• Decision it can influence.

Avoid targets without a baseline. First establish clean measurement and enough volume for a responsible comparison.

7.3 Review as a sequence

A low product click rate can come from weak offer fit, unclear teaching, a broken link, or the wrong entry audience. Review the path from delivery to decision. Do not rewrite the final CTA while ignoring earlier mismatch.

8.1 Write a decision-first hypothesis

Use:

For [eligible audience], changing [one variable] from [control] to [variant] may affect [primary metric] because [reason]. We will keep [important constants] fixed and use the result to decide [specific action].

Example:

For new flow-planner subscribers, changing Message 3 from a long explanation to a three-action worksheet walkthrough may affect planner clicks because the action becomes easier to recognize. We will keep timing, sender, and destination fixed and use the result to decide which teaching format to keep.

8.2 Test meaningful variables

Useful test categories:

• Message premise.
• Sequence order.
• Teaching format.
• CTA wording and destination.
• Timing after the subscriber action.
• Segmentation based on declared need.
• Offer explanation.

Cosmetic tests can be valid, but they rarely repair a broken journey.

8.3 Protect interpretation

• Change one major variable at a time.
• Define eligibility and exclusions before launch.
• Record the start and stop rule.
• Note promotions, outages, list-source changes, or seasonality.
• Do not repeatedly inspect and stop only when a preferred result appears.
• Treat small samples as directional.
• Keep the control when evidence is inconclusive.

8.4 Maintain an experiment record

The experiment tracker should contain the hypothesis, audience, variable, control, variant, metric definition, guardrails, dates, observations, limitations, decision, and follow-up.

A "losing" variant can still produce useful knowledge if the setup and decision are documented.