Measuring payment recovery with a holdout group
Every recovery tool can show you a list of invoices that were paid after it sent an email. The question that matters is different: how many of those would have been paid anyway? A holdout group is the simplest way to answer it, and the only one that does not depend on trusting the vendor.
Why 'paid after our email' is not a result
When a payment fails, several things happen at once. Stripe retries the card. Stripe may email the customer. The customer may notice on their own when they check their bank app or try to log in. A card network may send an updated card number. Any of these can lead to a paid invoice, with or without your follow-up.
So if you count every invoice paid after a follow-up as recovered by the follow-up, you are counting part of what Stripe would have recovered alone. The number looks good, and it is wrong in a direction that always flatters the tool.
Why before-and-after comparisons mislead too
A second common approach compares recovery this quarter with recovery last quarter. It is better than nothing, but many things change between two periods: the season, the customer mix, prices, Stripe's own retry logic, the share of annual plans renewing. A rise in recovery after switching on a tool may have little to do with the tool.
How a holdout group works
The idea comes from clinical trials and marketing experiments. Each time a payment fails, a random draw decides whether the invoice goes into the treatment group, which receives your follow-ups, or into the holdout group, which receives none. Everything else stays the same for both: Stripe keeps retrying, and Stripe's own emails, if you use them, still go out.
Because the draw is random, the two groups are comparable on average: same kinds of customers, same decline reasons, same amounts. After a fixed window, you compare the share of failed amounts that got paid in each group. The difference is what your follow-ups added.
lift = (treatment recovery rate − holdout recovery rate) × failed amount in treatment
Recovery rates are best computed on amounts rather than counts, so that one large invoice weighs as much as it should, and always per currency. On a quiet month the difference can even be negative; report it as it is instead of hiding it.
Choosing the details
- Size of the holdout: large enough to be measurable, small enough to cost little. Five to ten percent of failed invoices is a common choice. The holdout is a cost: those invoices do not get your follow-ups.
- The draw: it must be decided before any action and must not depend on the customer, the amount or the decline. A hash of the invoice identifier, compared with a threshold, is simple and can be checked by anyone afterwards.
- The window: decide in advance how long an invoice has to be paid to count as recovered, for example 30 days after the first failure. Without a fixed window, results drift as invoices age.
- Minimum volume: with a handful of holdout invoices, the holdout rate jumps around. Until the holdout reaches a reasonable size, a historical rate measured before the tool started is a safer reference.
Reading the result
Suppose that over a quarter, in one currency, the treatment group recovers a larger share of its failed amount than the holdout. Multiply the difference in rates by the treatment group's failed amount and you have the revenue your follow-ups added in that currency over that period.
- If the difference is close to zero, your follow-ups are not adding much on top of what already happens. That is useful to know.
- If it is negative, something in your follow-ups may be hurting, for example a tone that pushes customers to cancel. Look at the emails before anything else.
- If it is positive, keep measuring. One quarter with few invoices can be luck; several quarters in the same direction are evidence.
Common mistakes
- Choosing the holdout by hand, for example keeping small invoices aside. The groups stop being comparable.
- Moving an invoice from the holdout to the treatment group because the amount is large. The draw must be final.
- Adding currencies together, or converting at today's rate and summing.
- Counting refunds and disputes as recovered. A payment refunded a few weeks later was not really recovered; subtract it.
- Changing the window or the formula after seeing the numbers.
Each of these mistakes tends to make the result look better than it is. That is why the rules should be written down before the first invoice is drawn.
Why it matters when you choose a tool
Recovery tools are usually judged, and often priced, on what they recover. If 'recovered' includes what Stripe would have brought back on its own, the figure overstates the tool, whatever it costs. With a holdout, you see the difference it really made and can decide whether that is worth its price.
Reclaiva is built around this. The draw is deterministic and recomputable from the invoice identifiers, the holdout never receives anything, the formula is the one written in the terms, and the monthly statement lists every invoice behind the lift. When the holdout is still too small, the reference is the historical rate measured during the free audit. For the full calculation of what you lose today, read how much do failed payments cost you.