If you implement scoring rules for identifying at-risk customers, you will want to be sure the rules actually help your customer success organization, i.e., that the rules provides a return on investment (ROI). A scoring rule with low accuracy hurts your organization by reducing efficiency and effectiveness. Many scoring rules are developed from the experience and intuition of customer success managers, but most have not been validated as to their customer coverage or predictive accuracy – the two facets that determine ROI in predicting churn.
What are these facets and how do they affect your organization?
Customer coverage is the percentage measurement of the customer base that the scoring rule provides insight. For example if a rule is implemented based on survey response there will be low coverage. Typically, 20 percent is considered an awesome response rate for a survey. Most survey response rates are in the 10-15 percent ranged. The implication is that more than 80 percent of you customers will not respond to the survey which means that scoring rule based on survey responses has low customer coverage. Creating rules around other metrics such as number of open support cases or attendance at customer events can fall into the low coverage category. Rules with high customer coverage include demographic, firmographic, purchase history, and usage metrics.
Predictive accuracy is the percentage measurement of a rule’s true versus false predictions. In the case of churn, if a scoring rule predicts someone to be a non-churner and they do in fact churn, that is a false prediction. Likewise another false prediction would be if the rule predicts a churner and without any intervention from customer success, the churner renews. True predictions are of course the opposite of these. The percentage of true predictions versus the total provides predictive accuracy.
So how do these metrics provide help discover ROI for a customer success team?
Customer success teams are most efficient and effective if they intervene only when needed – i.e., if a customer is at risk. Interventions with customers not at risk are not only an unwarranted expense but also represent opportunity costs of assisting the actual at-risk customers. Therefore, low predictive accuracy actually lowers customer success organization’s efficiency and effectiveness. And low customer coverage, means missed opportunity.
The dynamics of customer coverage and predictive accuracy are shown in the following graphic. 
If the metric used has low customer coverage and predictive accuracy, customer success teams will be constantly reacting to fire drills in order to save customers. If the metric has high coverage and low accuracy, the customer success team will have lots of false alarms resulting in wasted effort and opportunity costs. Likewise high accuracy and low coverage will result in unanticipated churn. Only with high accuracy and high coverage can you get to efficient and effective churn prevention.
The take away is that predictive analytics is a powerful tool for creating an efficient and effective customer success organization. With that said, predictive analytics developed from experience and intuition without validation can actually have the opposite effect.
