15.3. Logistic Regression#

Hide code cell source

path_data = '../../data/'

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import accuracy_score, confusion_matrix, precision_score, recall_score

# The shopping session data and the train-test split from Section 1501.
shoppers = pd.read_csv(path_data + 'online_shoppers.csv')
shoppers['purchase'] = (shoppers['outcome'] == 'purchase').astype(int)
features = ['page_value', 'exit_rate', 'product_pages']
X = shoppers[features]
y = shoppers['purchase']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=14, stratify=y)

# Error costs from Section 1502.
COST_FN = 30     # dollars lost for each buyer the model misses (an assumption)
COST_FP = 3      # dollars wasted for each non-buyer the model flags (an assumption)

def error_cost(actual, predicted):
    """Total dollar cost of the false negatives and false positives."""
    tn, fp, fn, tp = confusion_matrix(actual, predicted).ravel()
    return COST_FN * fn + COST_FP * fp

# The fitted models and their test-set probabilities, so the exercises also work
# when the cells above them have not been run. The cells below refit the same models.
logit = LogisticRegression().fit(X_train, y_train)
p_buy = logit.predict_proba(X_test)[:, 1]
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=11)).fit(X_train, y_train)
p_buy_knn = knn.predict_proba(X_test)[:, 1]

A nearest neighbor classifier answers “purchase” or “no purchase”. For many business decisions, a probability is more useful: a session with a 90% chance of ending in a purchase deserves a nudge right now, one with a 15% chance might get a cheaper one, and one with a 1% chance can be left alone. Section Evaluating Classifiers also suggested that when a missed buyer costs much more than a false alarm, it may pay to flag sessions whose chance of a purchase is below 50%. That requires a classifier that reports how likely each outcome is.

Logistic regression is the most widely used classifier of this kind. Despite its name, it is a classification method: it predicts the probability that an observation belongs to the positive class. It builds on the linear regression of Chapter 13, so its coefficients can be read in a similar way. This section fits it to the same real shopping session data (UCI Online Shoppers Purchasing Intention, Sakar and others, 2019; CC BY 4.0), training set, and test set as Sections Nearest Neighbors with scikit-learn and Evaluating Classifiers.

15.3.1. The Logistic Curve#

Linear regression predicts a number by adding up the features, each multiplied by a slope: \(z = b_0 + b_1 x_1 + b_2 x_2 + \cdots\). That number can be anything, such as \(-3\) or \(250\), but a probability must lie between 0 and 1. Logistic regression fixes this by passing \(z\) through the logistic function, also called the sigmoid:

\[p = \frac{1}{1 + e^{-z}}\]

The logistic function turns any number into a value between 0 and 1. When \(z = 0\), \(p = 0.5\); large positive \(z\) gives \(p\) close to 1, and large negative \(z\) gives \(p\) close to 0.

def logistic(z):
    return 1 / (1 + np.exp(-z))

z = np.linspace(-6, 6, 200)

fig, ax = plt.subplots(figsize=(6, 3.5))
ax.plot(z, logistic(z), lw=3)
ax.axhline(0.5, color='gray', lw=1, ls='--')
ax.set_xlabel('z (weighted sum of the features)')
ax.set_ylabel('probability p')
plt.show()
../../_images/c58aff40f28ceece7472f1afa8a1d42021f00a87d4dd551b2898760f50a4c942.png

Here is the curve in action with a single feature. Fit logistic regression to predict a purchase from the exit rate alone. Then group the training sessions by exit rate and compare the model’s predicted probability at each group’s average exit rate with the actual purchase rate in the group.

exit_model = LogisticRegression()
exit_model.fit(X_train[['exit_rate']], y_train)

bands = pd.cut(X_train['exit_rate'], [-0.001, 0.005, 0.01, 0.02, 0.03, 0.05, 0.1, 0.15, 0.3])
summary = pd.DataFrame({'exit_rate': X_train['exit_rate'], 'purchase': y_train}).groupby(bands, observed=True).agg(
    sessions=('purchase', 'size'),
    average_exit_rate=('exit_rate', 'mean'),
    actual_purchase_rate=('purchase', 'mean'))

by_band = summary.reset_index(drop=True)
by_band['predicted probability'] = exit_model.predict_proba(
    by_band[['average_exit_rate']].rename(columns={'average_exit_rate': 'exit_rate'}))[:, 1]
by_band.round(3)
sessions average_exit_rate actual_purchase_rate predicted probability
0 497 0.003 0.296 0.211
1 853 0.008 0.260 0.202
2 2041 0.015 0.230 0.188
3 1535 0.025 0.180 0.172
4 1696 0.040 0.096 0.149
5 1225 0.074 0.043 0.105
6 222 0.128 0.009 0.059
7 562 0.197 0.005 0.027
grid = pd.DataFrame({'exit_rate': np.linspace(0, 0.2, 100)})

fig, ax = plt.subplots(figsize=(6, 3.5))
ax.plot(grid['exit_rate'], exit_model.predict_proba(grid)[:, 1], lw=3, label='logistic regression')
ax.scatter(by_band['average_exit_rate'], by_band['actual_purchase_rate'], s=60, color='darkorange',
           zorder=5, label='actual purchase rate')
ax.set_xlabel('exit rate')
ax.set_ylabel('probability of a purchase')
ax.legend(loc='upper right')
plt.show()
../../_images/5dbee6e684a1257d023900e581dbb3e67ea61ded7a04429a95ed997e90f7f699.png

The predicted probability falls along an S-shaped curve as the exit rate rises, from about 0.21 at an exit rate near 0 to about 0.03 at 0.2. (Over this narrow range, only the right-hand part of the S shows.) The actual purchase rates fall in the same direction, but faster than the curve: the curve is too low for sessions with very low exit rates (0.21 predicted against 0.30 actual) and too high for sessions with high exit rates (0.03 predicted against 0.005). One feature alone gives a rough fit; the next subsection uses all three.

### Exercise: The Logistic Function
#   1. Write ex_logistic(z) = 1 / (1 + e^(-z)) using np.exp.
#   2. Apply it to the array ex_z and print the results rounded to 3 decimals.
# Print exactly:
# [0.018 0.269 0.5   0.731 0.982]
ex_z = np.array([-4, -1, 0, 1, 4])
### Your code starts here.




### Your code ends here.

Hide code cell source

### Solution
ex_z = np.array([-4, -1, 0, 1, 4])

def ex_logistic(z):
    return 1 / (1 + np.exp(-z))

print(ex_logistic(ex_z).round(3))
[0.018 0.269 0.5   0.731 0.982]

15.3.2. Fitting LogisticRegression#

Fitting logistic regression with all three features follows the same fit pattern as every other scikit-learn model. Behind the scenes, fit searches for the intercept and coefficients that make the observed purchase outcomes in the training set as likely as possible.

logit = LogisticRegression()
logit.fit(X_train, y_train)

coefficients = pd.DataFrame({'coefficient': logit.coef_[0].round(3)}, index=features)
print('Intercept:', logit.intercept_.round(3))
coefficients
Intercept: [-2.441]
coefficient
page_value 0.088
exit_rate -5.737
product_pages 0.006

Reading the coefficients. Each coefficient multiplies its feature inside \(z\), so its sign tells the direction of the effect, holding the other features fixed:

  • A positive coefficient means that larger values of the feature raise \(z\) and therefore raise the predicted probability of a purchase.

  • A negative coefficient means that larger values lower the probability.

Here page value and the number of product pages have positive coefficients (sessions that see valuable pages and view more products are more likely to end in a purchase), while the exit rate has a negative one (sessions on pages that visitors often leave are less likely to). These directions match the group averages in Section Nearest Neighbors with scikit-learn.

The size of a coefficient is harder to read than in linear regression, because the logistic curve is not a straight line: the same change in \(z\) moves the probability a lot near 50% and only a little near 0% or 100%. (Statisticians translate coefficients into changes in the odds of a purchase; that is beyond this introduction.) The coefficients also depend on each feature’s units, so do not compare their sizes across features unless the features have been scaled. The exit rate’s coefficient, \(-5.737\), looks huge next to \(0.006\) for product pages, but the exit rate only ranges from 0 to 0.2 while the page count runs to 705; each extra 0.1 of exit rate lowers \(z\) by about 0.57, and each extra 100 product pages raises it by about 0.6.

The clearest way to see what the model says is to compare predicted probabilities for sessions that differ in one feature. Here are five sessions with an exit rate of 0.02 and 30 product pages, whose pages have page values from $0 to $40.

what_if = pd.DataFrame({'page_value': [0.0, 10.0, 20.0, 30.0, 40.0],
                        'exit_rate': [0.02] * 5,
                        'product_pages': [30] * 5})
what_if['probability of purchase'] = logit.predict_proba(what_if[features])[:, 1].round(3)
what_if
page_value exit_rate product_pages probability of purchase
0 0.0 0.02 30 0.086
1 10.0 0.02 30 0.184
2 20.0 0.02 30 0.352
3 30.0 0.02 30 0.566
4 40.0 0.02 30 0.758

A session whose pages have a page value of $0 has a predicted probability of purchase of only 9%, and $40 raises it to 76%. The steps are uneven: going from $0 to $10 raises the probability by 10 points, from $20 to $30 by 21 points, and from $30 to $40 by 19 points. How much a feature changes the probability depends on where the session sits on the curve.

### Exercise: A Two-Feature Model
#   1. Fit a LogisticRegression on X_train[['page_value', 'exit_rate']] and y_train.
#   2. Print the two coefficients rounded to 3 decimals.
#   3. Print whether each one raises or lowers the probability of a purchase.
# Print exactly:
# page_value: 0.087 (raises purchase probability)
# exit_rate: -6.902 (lowers purchase probability)
### Your code starts here.




### Your code ends here.

Hide code cell source

### Solution
two_features = ['page_value', 'exit_rate']
model2 = LogisticRegression()
model2.fit(X_train[two_features], y_train)

for name, coef in zip(two_features, model2.coef_[0]):
    direction = 'raises' if coef > 0 else 'lowers'
    print(f'{name}: {round(coef, 3)} ({direction} purchase probability)')
page_value: 0.087 (raises purchase probability)
exit_rate: -6.902 (lowers purchase probability)

15.3.3. Predicted Probabilities#

For each session, the fitted model’s predict_proba method returns one column per class: the probability of class 0 (no purchase) and of class 1 (purchase). Each row adds up to 1. You usually keep the second column, the probability of the positive class.

proba = logit.predict_proba(X_test)
print(proba[:5].round(3))

p_buy = proba[:, 1]
[[0.921 0.079]
 [0.973 0.027]
 [0.878 0.122]
 [0.134 0.866]
 [0.964 0.036]]
test_view = X_test.copy()
test_view['p_buy'] = p_buy.round(3)
test_view['predicted'] = logit.predict(X_test)
test_view['actual'] = y_test
test_view.head(8)
page_value exit_rate product_pages p_buy predicted actual
5218 0.000000 0.016667 12 0.079 0 0
421 0.000000 0.200000 1 0.027 0 0
5110 5.694121 0.042323 34 0.122 0 0
6614 45.711463 0.009091 55 0.866 1 1
1817 0.000000 0.150000 4 0.036 0 0
5654 0.000000 0.111111 9 0.046 0 0
4549 0.000000 0.026500 34 0.085 0 0
12122 0.000000 0.021839 231 0.245 0 0

The predict method turns each probability into a label using a cutoff of 0.5: “purchase” when the probability is above 0.5, otherwise “no purchase” (a probability of exactly 0.5 almost never occurs). You can check that this is exactly what it does.

print('predict matches p_buy > 0.5:', np.array_equal(logit.predict(X_test), (p_buy > 0.5).astype(int)))
print('Test accuracy:', round(logit.score(X_test, y_test), 3))
predict matches p_buy > 0.5: True
Test accuracy: 0.885

A histogram of the predicted probabilities, split by what actually happened, shows how well the model separates the two groups. The vertical axis is on a log scale so that the small bars can be seen. Sessions without a purchase mostly get low probabilities, below 0.2. Buyers are split: some get probabilities near 1, but many get low ones, and only 222 of the 572 buyers (39%) are above the 0.5 line.

fig, ax = plt.subplots(figsize=(6, 3.5))
bins = np.arange(0, 1.05, 0.05)
ax.hist(p_buy[y_test == 0], bins=bins, alpha=0.6, label='no purchase', ec='white')
ax.hist(p_buy[y_test == 1], bins=bins, alpha=0.6, label='purchase', ec='white')
ax.axvline(0.5, color='k', lw=2, ls='--')
ax.set_xlabel('predicted probability of a purchase')
ax.set_ylabel('test sessions (log scale)')
ax.set_yscale('log')
ax.legend()
plt.show()
../../_images/37e03efe8685424f01d335d2de37654a93a9b6a937cbf703b4f00c7e09d7a2ed.png
### Exercise: Likely Buyers
#   1. Using p_buy (the test-set purchase probabilities), count the test sessions
#      whose predicted probability of a purchase is above 0.7.
#   2. Among those sessions, compute the share that actually ended in a purchase
#      (use y_test), rounded to 3 decimals.
# Print exactly:
# Likely buyers: 214
# Share who bought: 0.776
### Your code starts here.




### Your code ends here.

Hide code cell source

### Solution
likely_buyers = p_buy > 0.7

print('Likely buyers:', likely_buyers.sum())
print('Share who bought:', round(y_test[likely_buyers].mean(), 3))
Likely buyers: 214
Share who bought: 0.776

15.3.4. Changing the Decision Threshold#

The cutoff of 0.5 is called the decision threshold. Nothing forces you to use 0.5. Lowering the threshold flags more sessions: recall goes up (more buyers are caught) and precision usually goes down (more false alarms). Raising it does the opposite. Which threshold is best depends on the costs from Section Evaluating Classifiers: a missed buyer costs $30 and a false alarm $3, both assumptions made for that example.

rows = []
for threshold in [0.05, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5]:
    flagged = (p_buy >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_test, flagged).ravel()
    rows.append({'threshold': threshold,
                 'flagged': tp + fp,
                 'FP': fp, 'FN': fn,
                 'accuracy': round(accuracy_score(y_test, flagged), 3),
                 'precision': round(precision_score(y_test, flagged), 3),
                 'recall': round(recall_score(y_test, flagged), 3),
                 'error cost ($)': error_cost(y_test, flagged)})

thresholds = pd.DataFrame(rows)
thresholds
threshold flagged FP FN accuracy precision recall error cost ($)
0 0.05 3279 2709 2 0.267 0.174 0.997 8187
1 0.10 1154 657 75 0.802 0.431 0.869 4221
2 0.15 789 354 137 0.867 0.551 0.760 5172
3 0.20 627 240 185 0.885 0.617 0.677 6270
4 0.30 476 160 256 0.888 0.664 0.552 8160
5 0.40 370 102 304 0.890 0.724 0.469 9426
6 0.50 298 76 350 0.885 0.745 0.388 10728
fig, ax = plt.subplots(figsize=(6, 3.5))
ax.plot(thresholds['threshold'], thresholds['error cost ($)'], marker='o', lw=2)
ax.set_xlabel('decision threshold')
ax.set_ylabel('total error cost ($)')
plt.show()
../../_images/2efebe4dd10de21e2abf0788a0eed63e81f28c282f37f9b4ae77f0085eee093f.png

At the default threshold of 0.5, the errors cost $10,728, more than the $9,381 of flagging every session (Section Evaluating Classifiers), because only 39% of the buyers are caught. Lowering the threshold to 0.1 flags 1,154 sessions instead of 298, raises recall from 0.39 to 0.87, and cuts the error cost to $4,221, a saving of more than $6,500 on 3,699 sessions, even though accuracy falls from 0.885 to 0.802. The retailer sends more nudges that turn out to be unnecessary, but each one is cheap compared with a lost sale. Going too far does not help: at 0.05 almost every session is flagged (3,279 of 3,699) and the cost rises again to $8,187. Notice also that the most accurate threshold (0.4, with accuracy 0.890) costs $9,426, more than twice the cost at 0.1.

There is a simple rule of thumb behind this. Flag a session when the expected cost of not flagging is larger than the cost of flagging:

\[30 \times p > 3 \times (1 - p) \quad\Longleftrightarrow\quad p > \frac{3}{3 + 30} \approx 0.09 .\]

If the probabilities are accurate, the cost-minimizing threshold is the false-alarm cost divided by the sum of the two costs, about 0.09 here, which is close to the 0.1 that the table shows to be cheapest. As with \(k\) in Section Nearest Neighbors with scikit-learn, a careful analysis would choose the threshold on a validation set and confirm it on the test set.

### Exercise: A Threshold of 0.25
#   1. Flag the test sessions whose p_buy is at least 0.25.
#   2. Print the number flagged, the recall rounded to 3 decimals, and the
#      total error cost with error_cost(y_test, ex_flagged).
# Print exactly:
# Flagged: 542
# Recall: 0.607
# Error cost: 7335
### Your code starts here.




### Your code ends here.

Hide code cell source

### Solution
ex_flagged = (p_buy >= 0.25).astype(int)

print('Flagged:', ex_flagged.sum())
print('Recall:', round(recall_score(y_test, ex_flagged), 3))
print('Error cost:', error_cost(y_test, ex_flagged))
Flagged: 542
Recall: 0.607
Error cost: 7335

15.3.5. Comparing with Nearest Neighbors#

A nearest neighbor classifier (kNN, short for \(k\)-nearest neighbors) can also report a probability: predict_proba gives the share of the \(k\) neighbors in each class. With \(k = 11\), the possible probabilities are \(0/11, 1/11, \ldots, 11/11\). So you can compare the two classifiers on the same test set, both at the usual threshold of 0.5 and at a lower threshold suited to the costs.

knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=11))
knn.fit(X_train, y_train)
p_buy_knn = knn.predict_proba(X_test)[:, 1]

def summarize(name, probabilities, threshold):
    flagged = (probabilities >= threshold).astype(int)
    return {'classifier': name, 'threshold': threshold,
            'accuracy': round(accuracy_score(y_test, flagged), 3),
            'precision': round(precision_score(y_test, flagged), 3),
            'recall': round(recall_score(y_test, flagged), 3),
            'error cost ($)': error_cost(y_test, flagged)}

pd.DataFrame([summarize('kNN (k = 11)', p_buy_knn, 0.5),
              summarize('logistic regression', p_buy, 0.5),
              summarize('kNN (k = 11)', p_buy_knn, 0.1),
              summarize('logistic regression', p_buy, 0.1)])
classifier threshold accuracy precision recall error cost ($)
0 kNN (k = 11) 0.5 0.888 0.667 0.554 8124
1 logistic regression 0.5 0.885 0.745 0.388 10728
2 kNN (k = 11) 0.1 0.811 0.441 0.830 4713
3 logistic regression 0.1 0.802 0.431 0.869 4221

At the usual threshold of 0.5 the two classifiers are close in accuracy (0.888 for nearest neighbors against 0.885), but nearest neighbors catches many more buyers (recall 0.554 against 0.388) and so has the lower error cost ($8,124 against $10,728). At a threshold of 0.1, which the cost rule suggests, both become much cheaper, and logistic regression is the cheaper of the two ($4,221 against $4,728). Both then beat the $9,381 of flagging every session. So neither method wins everywhere: the exercise below shows nearest neighbors ahead at a threshold of 0.2. On this real data set the two methods are close, and the choice of threshold matters more than the choice of method. Logistic regression draws one smooth curve in each feature, which may not suit a feature such as page value, where most sessions sit at exactly 0 (Section Classification).

Beyond the numbers, the two methods have different strengths:

  • Logistic regression gives smooth probabilities, coefficients whose signs explain the direction of each feature’s effect, and fast predictions. It assumes that the probability changes smoothly and in one direction as each feature grows.

  • Nearest neighbors makes no such assumption and can follow irregular patterns, but it offers no coefficients to explain its predictions, needs scaled features, and must search the whole training set for every prediction.

In practice, analysts often try both, compare them on validation data with the measure that matters to the business, and choose a threshold to match the costs.

### Exercise: Compare at a Threshold of 0.2
#   1. Flag test sessions at a threshold of 0.2, once with p_buy (logistic regression)
#      and once with p_buy_knn (11-nearest neighbors).
#   2. Print the error cost of each with error_cost(y_test, ex_flagged), and the name
#      of the cheaper classifier.
# Print exactly:
# logistic regression cost: 6270
# kNN cost: 5079
# Cheaper: kNN
### Your code starts here.




### Your code ends here.

Hide code cell source

### Solution
ex_costs = {}
for name, probabilities in [('logistic regression', p_buy), ('kNN', p_buy_knn)]:
    ex_flagged = (probabilities >= 0.2).astype(int)
    ex_costs[name] = error_cost(y_test, ex_flagged)
    print(name, 'cost:', ex_costs[name])

print('Cheaper:', min(ex_costs, key=ex_costs.get))
logistic regression cost: 6270
kNN cost: 5076
Cheaper: kNN