Impact Sizing

Impact Sizing or Impact Estimation is what product data scientists rigorously work on during the planning season. It helps answer key questions like:

1. How much incremental value/conversions will this project provide?
2. Should we prioritize project A or B?
3. Which ideas/features are worth pursuing?
4. Adding up all, can it help us meet the tops-down goal?

The 3 key variables for impact sizing are:

1. Audience Size: How many users does the new feature touch on? This is often easy to understand based on the app/surface the feature is launching on. Use weekly or monthly unique user count. If annual impact, is required, you will need to find the multiplier.
2. Baseline Conversion: What is the current conversion rate for the above audience before the launch? Use 7/14/30 day conversion rate as needed.
3. Estimated Lift: How much increase in conversion will your new feature drive? This is the tricky part and often needs domain expertise. Use past projects to estimate this or you can start with a flat rate or range like 2% to 10%, making your impact a range. You can refine it as you gather new data points.

Putting it all together:

Estimated Impact = Audience Size * Baseline Conversion * Estimate Lift

This provides absolute # of conversion the new feature will drive.

Sometimes it doesn’t stop there. You will need to provide a t-shirt size. It is easier to convey this by converting impact into % impact. You can do that by dividing it by the total # of conversions for your company or org in the specific period.

Impact % = Estimated Impact / Total Conversions

Then define bins like below and tag it to your projects accordingly:

<2% Lift = Small

2–5% Lift = Medium

5%+ Lift = Large

Impact sizing is part science, part art. You will master it the more time you spend on your domain.