Why our churn model is a logistic regression

When we started building the churn scorer, the question was not which algorithm can get the highest AUC on our training set. The question was which approach produces a score that a customer-success lead can act on in front of a customer, without needing to take it on faith.

Those are different questions, and the second one rules out more approaches than you might expect.

What we tried first

The standard modern path for a classification problem like churn prediction is to throw a gradient boosting model at it, tune it with XGBoost or LightGBM, and see what you get. On our labeled behavioral cohort, we got a held-out AUC of 0.943. We also tried a small neural network, two hidden layers, trained on the same features. AUC 0.941. Both beat the logistic regression, which landed at 0.938.

That is a real gap. In a holdout test of around 4,000 accounts, the difference between 0.938 and 0.943 is roughly two accounts per hundred that the better model ranks correctly. That matters.

Why we did not ship the better model

The issue with gradient boosting and neural networks for this use case is interpretability, and not in the abstract way the word is usually invoked. Specifically: when Churnora shows a customer-success lead that an account is at high risk because of declining sign-in frequency and seat underutilization, that lead needs to be able to say to their customer: you were logging in every day in January and now you are in once a week, and eight of your twelve seats have not been touched in 60 days. This is what we saw and this is why we called.

A tree ensemble does not have coefficients. It does not have a clean mapping from input features to output probability that you can summarize in two sentences. Its factors are a post-hoc approximation of what it actually computed, not the thing itself. That approximation is fine for a lot of applications. It is not fine when the factor is going into a call with a customer who will reasonably ask how you know.

Trust is the product. A model that cannot be explained to a customer-success lead is a model they will not act on, however good the AUC is. Two points of AUC for zero points of trust is a bad trade.

How we engineer the features

The model has eight input features, all behavioral: sign-in recency, sign-in volume over 30 days, the month-over-month trend in sign-in volume, feature breadth, seat utilization, support tickets filed in the last 90 days, payment failures in the last 90 days, and tenure in months.

Raw values go through two preprocessing steps. Log transforms handle the extreme right tails on features like tenure and ticket counts: a tenure of one month versus six months is a much bigger behavioral difference than 24 months versus 29 months, and the log scale reflects that. Then everything is standardized to mean zero, unit variance before fitting, so the coefficient magnitudes are comparable and the regularization parameter acts predictably.

The result is eight features with eight interpretable coefficients. Sign-in recency has the largest negative coefficient: accounts that have not signed in recently are much more likely to churn, everything else equal. Payment failures have the sharpest effect per unit increase. Tenure has a negative coefficient: older accounts churn at lower rates, and the relationship flattens at around 18 months.

What AUC 0.938 means, and does not mean

AUC measures how often the model ranks a randomly chosen churner above a randomly chosen non-churner. At 0.938, it does this correctly about 94 times in a hundred. On our cohort with a 15.3% base churn rate, the model is genuinely useful for ranking: the top decile of scores contains a much higher concentration of actual churners than the bottom decile.

What AUC does not mean: that every high-scoring account will churn. The model produces probability estimates. We calibrate using Platt scaling so the output probabilities are meaningful, not just ordinal rankings. At a score of 0.60, roughly 60% of accounts at that score in the holdout actually churned. An account scored at 0.25 is worth watching, not panicking about.

We chose calibration over the last point of AUC. A model that is honest about its uncertainty is a model people will keep using when it is occasionally wrong.

Why the language model does not touch the prediction

The only thing the language model does in Churnora is turn the model's top factors into readable sentences and draft the outreach. It does not see the raw data, it does not influence the score, and it cannot disagree with the scorer.

This is intentional. A language model is a plausible-sounding machine. If it could say this account looks fine actually, it would sometimes say that about an account the behavioral model correctly ranked at the top of the risk list. We do not want plausibility competing with the signal. The scorer scores. The language layer translates.

The factors you see in Churnora trace back to numbers in that account's row. Not to a language model's read of the situation.

Priya Natarajan

0.938

held-out AUC on our behavioral cohort, calibrated with Platt scaling so the output probabilities reflect the real base rate of 15.3%

8

engineered behavioral features: sign-in recency, sign-in volume, usage trend, feature breadth, seat utilization, support tickets, payment failures, tenure

0

language model influence on the churn prediction: the scorer runs entirely in our own code