The Right Goal Against the Right Metric

How do you measure the success of your product? Hopefully it isn't against the success of the company, or of even the larger org your product fits into, because they don't measure the success of your product - the one you have ownership of. Whether or not you're required to by your management, you need to pick a single or composite metric that measures the success of what you do - your business within the business. If you know what this metric is, then many of the decisions you have to make day-to-day as a product manager will be a snap.

Let's say you run the spam prevention team for a major webmail product. Your goal is "Reduce spam on the abcmail network", right? Wrong. First, because there are no numbers associated with the goal, and second because the phrase 'reduce spam' is too nebulous. Are you trying to reduce spam messages sent to abcmail account holders, or from them (that affects your mail reputation with other ISP's)? Do you care how many get sent to account holders, or only how many reach their inboxes? All account holders or only active ones? Is reducing any of these even possible, given a general increase in the volume of spam and increase in the abcmail account base?

If you think your goal is to reduce spam on the network then you'll come up with some ideas to prevent chunks of spam, but that spam may not be the most valuable spam you can prevent from the perspective of your users or your business.

I'll focus on the inbound case where you have to ask yourself, what spam do our customers care about, and what negative side-effects does this spam have on our business? Is user retention the main thing we're concerned about, or do negative opinions about our service spread by our existing customers negatively affect user acquisition? Let's assume that, based on our data and user feedback, we're most concerned about active user retention. Webmail users probably don't close their accounts and send a note to headquarters indicating that spam was the cause of their account closure, so we'll assume that ceasing to send mail from the account is equivalent to account cancellation. I'll call this 'account abandonment'.

So now we're part way there. Our proto-goal is to "Reduce account abandonment caused by spam". This is a very different goal than simply reducing # of spam emails received. It also focuses on active accounts, by definition, rather than all accounts, providing additional focus. From here we have to put a metric on account abandonment due to spam. I'll hand-wave here and assume that we've developed a metric called 'Badmail Rate' that takes # of emails marked by the user as spam plus emails deleted without being read, divided by all email, and we apply this against individual accounts. When an account is abandoned and this metric is above a certain threshold (say, 50%) we chalk it up to spam, below this threshold we chalk it up to other factors. I've included email deleted without being read because I assume large segments of users never use the report spam features of their webmail services.

So now our sights are set on Badmail Rate. We can look at the distribution of badmail rates for all active users and see what the right end of that distribution looks like. Now our goal is a real focusing tool. We can model the different projects we have in mind (something I'll discuss in a future post) and look at which we estimate will have the greatest impact decreasing our badmail rate, or at least the number of users who are approaching our badmail threshold.

In the real world we'd also probably have goals against the amount of outbound spam sent from our service, and likely another against overall volume inbound, but the difference between having one goal and three, is that with three we can make intelligent choices about resource allocation based on the impact of projects on one or more of the goals.

Now I've never worked directly on a mail product and I could be talking out of my ass here, but I did find it a useful exercise applying the goal-setting technique I use against a different domain.

-@ianmcall