Logistic regression · B06 · Sunflower Intelligence

Before you start

You will be able to

Explain why a straight-line model is unsuitable for a yes/no outcome, and convert between probability, odds and log-odds.
Interpret logistic regression coefficients as changes in log-odds and as odds ratios, holding the other predictors fixed.
Calculate a predicted probability from fitted coefficients, in a worksheet and by hand.
Test coefficients with the Wald z test and nested models with the likelihood-ratio test.
Evaluate a classification at a chosen threshold against the base rate, with sensitivity and specificity.
State what an odds ratio from observational data can and can’t say about the effect of an action.

Review first

Multiple regression — Several predictors, each slope with the others held fixed.
Categorical predictors and interactions — Dummy variables and reference levels, used here for the café, the weekend and notifications.
Transformations and influence — Logs and exponentials, and reading a coefficient on a log scale as a multiplication.
Probability — Probability, the base rate and the complement rule.

Fictional case A fictional coffee business founded by Ana Lim. It starts as a catering and barista-training service, opens its own cafés and an online shop for beans, and plans a canned cold brew.

The idea

Fictional case

In October 2027, 225 people joined Lantern’s loyalty app at the two cafés, and 121 of them came back within 30 days. Ana wants to know who comes back: members who live or work nearby, members who turn on the app’s notifications, members whose first purchase was larger, members who joined at Millbrook or at a weekend. She also wants a way to pick out, soon after they join, the members least likely to return, so the cafés can try to win them back.

A yes/no outcome

Whether a member returned is a binary outcome: 1 for yes, 0 for no. Its average is the share who returned, 121 ÷ 225 = 53.8%, the base rate. What Ana wants to model is the probability of a 1 for members with given characteristics.

An ordinary regression of the 0/1 outcome on the five predictors, a linear probability model, is easy to fit and easy to read: each slope is a change in the probability. It also fails in two ways. Its fitted “probabilities” run from 0.118 to 1.025, and one member (a Millbrook member with notifications on and an $8.50 first ticket) gets a value above 1, which no probability can be. And its slopes are the same everywhere: a dollar more on the first ticket adds 0.09 to the probability whether a member’s chance was 0.2 or 0.95. Real probabilities can’t keep rising by the same step; near 0 and 1 they have to flatten.

A straight line through yes/no data runs below 0 and above 1; the logistic curve stays between them and flattens at both ends (schematic).

Odds and log-odds

Logistic regression models the probability through two changes of scale. The odds of an event are its probability divided by the probability that it doesn’t happen, $p/(1 - p)$ : the odds of returning are 121 ÷ 104 = 1.16, “about 7 to 6 on”. Odds run from 0 to infinity. Their natural log, the log-odds or logit, runs from minus to plus infinity, with 0 for a probability of one half; here $\ln 1.16 = 0.15$ . A quantity that can take any value is one a linear model can fit without predicting the impossible.

So the model is $\ln\big(p/(1-p)\big) = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k$ , and to get back to a probability you invert the two steps: $p = 1 / (1 + e^{-(\beta_0 + \beta_1 x_1 + \dots)})$ , the logistic curve. However large or small the right-hand side, p stays between 0 and 1, and it changes fastest when p is near one half.

Ana’s model

Logistic regression of returning within 30 days (225 October joiners)
Term	Coefficient	SE	z	p-value	Odds ratio (95% CI)
Intercept	−2.842	0.845	−3.36	0.001
Lives or works within a mile	0.339	0.297	1.14	0.253	1.40 (0.78 to 2.51)
Notifications on	0.926	0.291	3.18	0.001	2.52 (1.43 to 4.47)
First ticket (per $)	0.401	0.127	3.15	0.002	1.49 (1.16 to 1.92)
Joined at Millbrook	0.847	0.388	2.18	0.029	2.33 (1.09 to 4.98)
Joined at a weekend	−0.489	0.327	−1.50	0.135	0.61 (0.32 to 1.16)

Likelihood-ratio test against the model with no predictors: 23.3 on 5 df, p below 0.001. McFadden’s pseudo-R²: 0.075.

Each coefficient is a change in log-odds for a one-unit change in its predictor, holding the others fixed. That is hard to picture, so exponentiate it: $e^{b}$ is the odds ratio, the factor by which the odds are multiplied. Members with notifications on had $e^{0.926} = 2.52$ times the odds of returning of otherwise similar members with them off. Each extra dollar on the first ticket multiplied the odds by 1.49. Members who joined at Millbrook had 2.33 times the odds of those who joined at Harbourside with the same ticket, notifications and the rest.

An odds ratio is not a ratio of probabilities, and its effect on the probability depends on where you start. For a Harbourside member who lives further away, joined on a weekday with a $5 first ticket, turning notifications on in the model takes the predicted probability from 0.30 to 0.52, 22 points. For a nearby Millbrook member with an $8 ticket it takes it from 0.83 to 0.92, 10 points, with the same odds ratio of 2.52. Report odds ratios for the model and probabilities for the people.

Notifications: an association, not an effect

Members chose whether to turn notifications on. Those who did may be keener on the café in the first place, and keenness, not the notifications, could be why they came back. Raw, 62% of members with notifications on returned, against 43% with them off. The log can’t separate keenness from the notifications, so the odds ratio of 2.52 says that members with notifications on returned more, holding the other four predictors fixed. It does not say that switching a member’s notifications on would multiply their odds by 2.52. To learn that, Ana would need to turn notifications on for a random half of new members and compare.

In depth

Fitting by maximum likelihood

There are no residuals to square: each member either returned or didn’t. Instead, for any set of coefficients the model gives every member a probability, and the likelihood is the probability the model assigns to what actually happened: $\prod p_i^{y_i}(1 - p_i)^{1 - y_i}$ , p for each member who returned and 1 − p for each who didn’t. Maximum likelihood chooses the coefficients that make this as large as possible, or equivalently its log, $\ell = \sum [y_i \ln p_i + (1 - y_i)\ln(1 - p_i)]$ . There is no formula; software searches iteratively (iteratively reweighted least squares) until the coefficients stop changing. Ana’s model reaches $\ell = -143.67$ ; a model with only an intercept, which gives every member the base rate, has $\ell_0 = -155.32$ .

Testing a coefficient: the Wald z

Each coefficient comes with a standard error, and z = b ÷ SE is compared with the standard normal distribution (not t: the theory is large-sample). For notifications, z = 0.926 ÷ 0.291 = 3.18, p = 0.001. The 95% interval for the coefficient is b ± 1.96 SE, and exponentiating its ends gives the odds ratio’s interval, 1.43 to 4.47; it is not symmetric about 2.52.

The coefficient for living nearby, 0.339, has p = 0.25 and an odds ratio interval from 0.78 to 2.51. That is not evidence that distance doesn’t matter: the fictional data were in fact built with a log-odds effect of 0.9 for living nearby (an odds ratio of 2.46), inside the interval. With 225 members the estimate is too imprecise to tell an odds ratio of 1 from one of 2.5. A non-significant coefficient means the data can’t pin it down.

Testing models: the likelihood-ratio test

To compare a model with a simpler one nested inside it, use twice the gain in log-likelihood: $G = 2(\ell_{\text{full}} - \ell_{\text{reduced}})$ , compared with a chi-square distribution whose degrees of freedom are the number of coefficients dropped. Against the intercept-only model, $G = 2(-143.67 + 155.32) = 23.3$ on 5 df, p below 0.001: the five predictors together say something. Dropping notifications alone gives G = 10.46 on 1 df, p = 0.001, close to the Wald test’s verdict. When the two disagree, prefer the likelihood-ratio test, which behaves better when coefficients are large or samples small. It is the logistic counterpart of the nested F test.

McFadden’s pseudo-R², $1 - \ell/\ell_0$ , is 0.075 here. It is not a share of variance, and values that look small are usual for individual yes/no outcomes; it is most useful for comparing models fitted to the same data.

Confounding shows up here too

Fitted on its own, the first ticket has an odds ratio of 1.21 per dollar (p = 0.049); with the café and the other predictors held fixed it is 1.49. Millbrook members spend less on a first visit ($4.23 on average, against $6.03 at Harbourside) yet returned more often (55% against 51%). Pooled, the low-spending, often-returning Millbrook members dilute the link between spending and returning. This is the same omitted-variable logic as in least squares.

From probabilities to decisions: a threshold

To classify members as likely or unlikely to return, choose a threshold: predict “returns” when the fitted probability is at least that value. Each choice gives a classification table of four counts: true positives (predicted to return and did), false positives, true negatives and false negatives.

Classification of the 225 October joiners at three thresholds (the model’s own data)
Threshold	Accuracy	Sensitivity	Specificity	Predicted to return
0.4	61.8%	86.0%	33.7%	173
0.5	63.6%	72.7%	52.9%	137
0.6	60.4%	49.6%	73.1%	88

Sensitivity: the share of those who returned that the model flags. Specificity: the share of those who didn’t that it clears.

Classification accuracy needs a benchmark. Predicting that every member returns is right 53.8% of the time, the base rate, without any model. At a threshold of 0.5 the model is right 63.6% of the time: better, but by 10 points, not 64. Raising the threshold trades sensitivity (catching those who return) for specificity (correctly clearing those who don’t), and the right balance depends on what each kind of error costs, not on maximising accuracy.

Assumptions and cautions

Linearity in the log-odds. Each numerical predictor’s effect on the log-odds is a straight line. Check by banding the predictor and comparing the observed log-odds in each band with the model’s, or by adding a squared term or a hinge as in the previous lesson.
Independent cases. Each member is one row. Members who joined together, or repeated rows for the same person, break this, as they would in least squares.
Enough of each outcome. Coefficients become unstable when a predictor almost perfectly separates the 0s from the 1s, or when there are few events per predictor; a common rule of thumb asks for at least 10 of the rarer outcome per predictor (here 104 non-returners for 5 predictors, about 21 each). It is a convention, not a law.
No error variance to check. The variance of a 0/1 outcome is set by its probability, p(1 − p), so there is no σ, no residual standard error and no equal-variance condition.

Formulas

Odds and log-odds

\text{odds} = \dfrac{p}{1 - p}, \qquad \operatorname{logit}(p) = \ln\dfrac{p}{1 - p}, \qquad p = \dfrac{\text{odds}}{1 + \text{odds}}

$p$: Probability of the event
$\text{odds}$: Odds of the event, from 0 to infinity
$\operatorname{logit}(p)$: Log-odds, from minus to plus infinity

In Excel: =B2/(1-B2) for the odds, =LN(B2/(1-B2)) for the log-odds.

The logistic regression model

\ln\dfrac{p}{1 - p} = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k, \qquad p = \dfrac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \dots + \beta_k x_k)}}

$p$: Probability that y = 1 for a case with these predictor values
$\beta_0$: Intercept: the log-odds when every predictor is 0
$\beta_j$: Change in log-odds for a one-unit rise in xⱼ, holding the others fixedper unit of xⱼ
$x_j$: Predictor jits own units
$k$: Number of predictors

Assumes. Independent cases; each predictor’s effect is linear on the log-odds scale.

Predicted probability in Excel: =1/(1+EXP(-(b0+b1*x1+b2*x2))).

Odds ratio and its interval

\text{OR} = e^{b}, \qquad e^{\,b \pm 1.96\,\text{SE}(b)}

$b$: A fitted coefficientlog-odds per unit of x
$\text{SE}(b)$: Its standard errorlog-odds per unit of x
$\text{OR}$: Factor by which the odds are multiplied for a one-unit rise in x

For a change of d units the odds ratio is e^(d·b). An OR of 1 means no association.

Wald test of a coefficient

z = \dfrac{b}{\text{SE}(b)}

$b$: A fitted coefficientlog-odds per unit of x
$\text{SE}(b)$: Its standard errorlog-odds per unit of x
$z$: Compared with the standard normal distribution

Two-sided p-value: =2*(1-NORM.S.DIST(ABS(z),TRUE)).

Likelihood-ratio test

G = 2\,(\ell_{\text{full}} - \ell_{\text{reduced}}), \qquad \text{df} = p_{\text{full}} - p_{\text{reduced}}

$\ell$: Maximised log-likelihood of each model
$p$: Number of coefficients in each model, counting the intercept
$G$: Test statistic, compared with the chi-square distribution on df degrees of freedom

Assumes. The reduced model is nested in the full one and both are fitted to the same rows.

p-value: =CHISQ.DIST.RT(G,df). McFadden’s pseudo-R² is 1 − ℓ ÷ ℓ₀, with ℓ₀ the intercept-only model’s.

Classification measures at a threshold

\text{accuracy} = \dfrac{TP + TN}{n}, \quad \text{sensitivity} = \dfrac{TP}{TP + FN}, \quad \text{specificity} = \dfrac{TN}{TN + FP}

$TP, FN$: Cases with y = 1 predicted 1 (true positives) and predicted 0 (false negatives)
$TN, FP$: Cases with y = 0 predicted 0 (true negatives) and predicted 1 (false positives)
$n$: Number of cases

Compare accuracy with the base rate of the commoner outcome, what always guessing it would score.

Worked examples

A first example: From coefficients to a probability

Fictional figures

Using Ana’s fitted model, find the predicted probability that a Millbrook member who doesn’t live nearby, joined on a weekday, turned notifications on and spent $4.50 on their first visit returns within 30 days.

Log-odds: $-2.842 + 0.339(0) + 0.926(1) + 0.401(4.50) + 0.847(1) - 0.489(0) = 0.736$ , using the unrounded coefficients.
Odds: $e^{0.736} = 2.09$ .
Probability: $2.09 / (1 + 2.09) = 0.676$ , the same as $1/(1 + e^{-0.736})$ .

Answer. About 0.68: roughly two chances in three of returning within 30 days.

What it means. Each step undoes one change of scale: the model adds up log-odds, the exponential turns them into odds, and odds ÷ (1 + odds) turns them into a probability. The same member with notifications off would have log-odds 0.736 − 0.926 = −0.190 and a probability of 0.45.

A harder example: Testing notifications two ways, and what the result means

Fictional figures

Is the association between notifications and returning statistically clear once the other four predictors are held fixed? Use the Wald test and the likelihood-ratio test (the model without notifications has ℓ = −148.90), give the odds ratio with its interval, and translate it into probabilities for a typical member.

Wald: z = 0.926 ÷ 0.291 = 3.18; p = 2 × (1 − Φ(3.18)) = 0.0015.
Likelihood ratio: $G = 2(-143.67 - (-148.90)) = 10.46$ on 1 df; =CHISQ.DIST.RT(10.46,1) = 0.0012. The two tests agree.
Odds ratio: $e^{0.926} = 2.52$ ; interval $e^{0.926 \pm 1.96 \times 0.291}$ = 1.43 to 4.47.
Probabilities: for a Harbourside member further away, joining on a weekday with a $5 ticket, the predicted probability is 0.30 with notifications off and 0.52 with them on; for the same member at Millbrook, 0.50 and 0.72.

Answer. Yes: z = 3.18 (p = 0.0015) and G = 10.46 (p = 0.0012). Odds ratio 2.52 (95% CI 1.43 to 4.47); about 22 points of probability for a member starting near 0.3 or 0.5.

What it means. The association is clear and large. What it isn’t is a measured effect of switching notifications on: members chose the setting, and whatever made them choose it may also make them return. The finding is a reason to test, by turning notifications on by default for a random half of November’s joiners, not a forecast of what doing so would achieve.

Applied: a business decision: Choosing whom to win back

Fictional figures

Ana wants the cafés’ staff to greet, by name, new members the model flags as unlikely to return, starting with those whose predicted probability is below a threshold. A threshold of 0.4 flags 52 of the 225 October joiners. How good is that list, compared with picking members at random?

Among the 52 members with predicted probability below 0.4, 35 did not return: 67%.
Among all 225, 104 did not return: 46%, the share a random list of 52 would catch on average (about 24 members).
So the list is about 1.5 times as rich in non-returners as a random one. It still misses most of them: 104 − 35 = 69 non-returners had predicted probabilities of 0.4 or more.
Raising the threshold to 0.5 flags 88 members, of whom 55 (63%) did not return: more non-returners caught, a less concentrated list.

Answer. At 0.4 the list has 52 members, 67% of them non-returners against 46% at random; at 0.5, 88 members, 63% non-returners.

What it means. The model ranks members usefully but not sharply: a McFadden R² of 0.075 is typical of models of individual choices. Which threshold to use depends on how many members the staff can greet well, not on accuracy. And these figures are on the model’s own data; Ana should check them on November’s joiners before relying on them.

Explore

Common mistakes

Fitting a straight line to a 0/1 outcome
A linear probability model can predict probabilities below 0 or above 1 (one October member gets 1.025) and assumes each predictor moves the probability by the same amount everywhere. Use logistic regression; if a linear model is used for a quick look, check its fitted range.
Reading an odds ratio as a ratio of probabilities
An odds ratio of 2.52 does not mean “2.52 times as likely”. With a starting probability of 0.30 it gives 0.52, 1.7 times as likely; with 0.83 it gives 0.92. Convert to probabilities for a stated kind of member.
Reading the coefficient as a change in probability
b is a change in log-odds. 0.926 for notifications is not 92.6 percentage points, nor 0.926 of anything a manager would recognise. Exponentiate it for the odds ratio, or work through the probabilities.
Judging the model by its accuracy alone
63.6% sounds modest and 90% would sound good, but neither means anything without the base rate. With 53.8% of members returning, always guessing “returns” scores 53.8%. Compare with that, and look at sensitivity and specificity.
“Notifications double the odds of returning, so turn them on for everyone.”
Members chose notifications themselves, so the odds ratio describes members who chose them, holding four other things fixed. It is not the effect of switching them on. Randomise to find that.
Treating a non-significant odds ratio as “no effect”
Living nearby has p = 0.25, but its interval runs from 0.78 to 2.51. With 225 members the data can’t distinguish no association from a strong one.

Guided practice

Work through these with hints when you need them. Every hint and every worked solution is there to be used.

Loading four questions…

On your own

No hints this time. Answer, check, and read the feedback; the solution appears once you have answered or if you ask for it. Your first answers count toward mastery.

Loading six questions…

In practice

Fictional case

Ana’s note to the café managers: “Of October’s 225 new members, 121 (54%) came back within 30 days. Holding the café, distance, first ticket and weekend fixed, members with notifications on had 2.5 times the odds of returning (95% CI 1.4 to 4.5), and each extra dollar on the first visit went with 1.5 times the odds. These are associations: members chose notifications, so we don’t yet know whether turning them on would help. In November, new members at both cafés will be offered notifications on by default on alternate days, so we can measure it. Meanwhile, staff will greet by name the members the model gives under a 40% chance of returning.”

Logistic regression is one of the most used models in business analytics because so many business outcomes are yes or no: a customer renews or cancels, an applicant repays or defaults, a visitor buys or leaves, an invoice is paid on time or late. In each case the work splits as Ana’s did. One use is explanation, which predictors go with the outcome and by how much, reported as odds ratios with intervals and translated into probabilities for recognisable kinds of customer. The other is ranking: a predicted probability for every customer, a threshold, and a list.

The two uses need different checks. For explanation, the questions are those of every observational regression: what was left out, what was chosen by the customers themselves, which comparisons are fair. For ranking, the questions are about new data: does the model rank next month’s customers as well as it ranked the ones it was fitted to, and is the list better than the simple rule it replaces? A model can be good for one use and poor for the other, and saying which is part of reporting it.

The most valuable step in Ana’s note is the last but one. An observational odds ratio cannot settle whether an action works, but it can point to the action worth testing, and a small randomised trial then answers the question directly. Turning a model’s association into a test is often worth more than refining the model.

Knowledge check

Six questions, one attempt each, no hints. Score 80% or more to complete the lesson; you can take it again at any time.

Loading six questions…

Summary

For a yes/no outcome, model the log-odds, ln(p ÷ (1 − p)), as a linear function of the predictors; the probability, 1 ÷ (1 + e^(−linear part)), then stays between 0 and 1. A straight line doesn’t (one October member gets 1.025).
A coefficient is a change in log-odds per unit of x, holding the others fixed; e^b is the odds ratio (notifications: 2.52, 95% CI 1.43 to 4.47). Its effect on the probability depends on the starting probability.
The model is fitted by maximum likelihood. Test a coefficient with the Wald z = b ÷ SE, and nested models with the likelihood-ratio G = 2(ℓ_full − ℓ_reduced) on a chi-square distribution (all five predictors: G = 23.3 on 5 df).
A non-significant coefficient with a wide interval means the data can’t pin the effect down, not that it is zero.
Classify with a threshold and judge the table against the base rate (63.6% accuracy at 0.5 against 53.8% for always guessing “returns”); choose the threshold from what each error costs, and check on new data.
From observational data an odds ratio is an association: notifications were self-selected, so randomise to learn the effect of turning them on.

Terms introduced

Odds: The probability that an event happens divided by the probability that it doesn’t, p ÷ (1 − p); odds of 3 mean the event is three times as likely to happen as not.
Log-odds: The natural logarithm of the odds, ln(p ÷ (1 − p)), also called the logit. It is 0 when p = 0.5 and can take any value, which is why logistic regression models it as a linear function of the predictors.
Odds ratio: The factor by which the odds of an outcome are multiplied between two groups, or for a one-unit rise in a predictor; in logistic regression, e raised to the coefficient. An odds ratio of 1 means no association.
Likelihood: The probability a model with given parameter values assigns to the data actually observed. Maximum likelihood estimation chooses the values that make it largest.
Classification accuracy: The share of cases a classification rule gets right: true positives plus true negatives over all cases. It should be compared with the base rate of the commoner outcome.
Sensitivity: The share of actual positive cases that a classification rule flags as positive, TP ÷ (TP + FN); also called recall or the true positive rate.
Specificity: The share of actual negative cases that a classification rule correctly classes as negative, TN ÷ (TN + FP).

Sources

Penn State Eberly College of Science. STAT 462: Applied Regression Analysis, notes on binary logistic regression (the model, likelihood, Wald and likelihood-ratio tests, odds ratios). online.stat.psu.edu/stat462/node/207
Penn State Eberly College of Science. STAT 501: Regression Methods, 15.3, “Further Logistic Regression Examples”. online.stat.psu.edu/stat501/book/export/html/1015
Microsoft Support. “LN function”. support.microsoft.com/en-us/Excel/functions/ln-function

Before you start

You will be able to

Review first

The idea

A yes/no outcome

Odds and log-odds

Ana’s model

Notifications: an association, not an effect

In depth

Fitting by maximum likelihood

Testing a coefficient: the Wald z

Testing models: the likelihood-ratio test

Confounding shows up here too

From probabilities to decisions: a threshold

Assumptions and cautions

Formulas

Odds and log-odds

The logistic regression model

Odds ratio and its interval

Wald test of a coefficient

Likelihood-ratio test

Classification measures at a threshold

Worked examples

Explore

Common mistakes

Fitting a straight line to a 0/1 outcome

Reading an odds ratio as a ratio of probabilities

Reading the coefficient as a change in probability

Judging the model by its accuracy alone

“Notifications double the odds of returning, so turn them on for everyone.”

Treating a non-significant odds ratio as “no effect”

Guided practice

On your own

In practice

Knowledge check

Summary

Key formulas

Terms introduced

Sources