14.3. Training, Testing, and Accuracy#
def standard_units(x):
return (x - np.mean(x))/np.std(x)
def standardize(df, columns, reference):
"""Convert columns of df to standard units using the mean and SD of reference.
Pass the training set as reference so that the test set never
influences how the features are scaled."""
scaled = df.copy()
for column in columns:
scaled[column] = (df[column] - reference[column].mean()) / reference[column].std(ddof=0)
return scaled
def distance(point1, point2):
"""The distance between two arrays of numbers."""
return np.sqrt(np.sum((point1 - point2)**2))
def all_distances(training, point):
"""The distance between point (an array of numbers) and each row of the training features."""
attributes = training.drop(columns=['Class'])
def distance_from_point(row):
return distance(point, np.array(row))
return attributes.apply(distance_from_point, axis=1)
def table_with_distances(training, point):
"""A copy of the training table with the distance from each row to point."""
training1 = training.copy()
training1['Distance'] = all_distances(training1, point)
return training1
def closest(training, point, k):
"""A table containing the k closest rows in the training table to point."""
with_dists = table_with_distances(training, point)
sorted_by_distance = with_dists.sort_values(by=['Distance'])
return sorted_by_distance.head(k)
def majority(topkclasses):
"""The most common class among the k nearest neighbors."""
return topkclasses.value_counts().idxmax()
def classify(training, point, k):
closestk = closest(training, point, k)
return majority(closestk['Class'])
def classify_grid(training, test, k):
ckd_new1 = pd.DataFrame(columns=['Hemoglobin', 'Glucose', 'Class', 'Distance'])
empty = np.array([])
for i in range(len(test)):
ckd_new2 = closest(training, np.array([test.iloc[i]]), k)
ones = len(ckd_new2[ckd_new2['Class'] == '1'])
zeros = len(ckd_new2[ckd_new2['Class'] == '0'])
if ones > zeros:
empty = np.append(empty, 1)
else:
empty = np.append(empty, 0)
return empty ### 0, 1
ckd = pd.read_csv(path_data + 'ckd.csv')
ckd.rename(columns={'Blood Glucose Random':'Glucose'}, inplace=True)
ckd['Class'] = ckd['Class'].astype(str)
features = ['Hemoglobin', 'Glucose', 'White Blood Cell Count']
color_table = pd.DataFrame(
{'Class':np.array([1, 0]),
'Color':np.array(['darkblue', 'gold']) ### darkblue == 1
}, index=np.array([1,0])
)
color_table['Class'] = color_table['Class'].astype(str)
# For the first plot only, we put the whole data set in standard units.
ckd_Standard_Unit = standardize(ckd[features + ['Class']], features, ckd)
ckd_combined = pd.merge(ckd_Standard_Unit, color_table, on='Class')
ckd_combined
| Hemoglobin | Glucose | White Blood Cell Count | Class | Color | |
|---|---|---|---|---|---|
| 0 | -0.865744 | -0.221549 | -0.569768 | 1 | darkblue |
| 1 | -1.457446 | -0.947597 | 1.162684 | 1 | darkblue |
| 2 | -1.004968 | 3.841231 | -1.275582 | 1 | darkblue |
| 3 | -2.814879 | 0.396364 | 0.809777 | 1 | darkblue |
| 4 | -2.083954 | 0.643529 | 0.232293 | 1 | darkblue |
| ... | ... | ... | ... | ... | ... |
| 153 | 0.700526 | 0.133751 | -0.569768 | 0 | gold |
| 154 | 0.978974 | -0.870358 | -0.216861 | 0 | gold |
| 155 | 0.735332 | -0.484162 | -0.601850 | 0 | gold |
| 156 | 0.178436 | -0.267893 | -0.409356 | 0 | gold |
| 157 | 0.735332 | -0.005280 | -0.537686 | 0 | gold |
158 rows × 5 columns
14.3.1. Overly Optimistic “Testing”#
The training set offers a very tempting set of patients on whom to test out our classifier, because we know the class of each patient in the training set.
But let’s be careful … there will be pitfalls ahead if we take this path. An example will show us why.
Suppose we use a 1-nearest neighbor classifier to predict whether a patient has chronic kidney disease, based on glucose and white blood cell count.
glucose_color_darkblue = ckd_combined[ckd_combined['Color'] == 'darkblue']
glucose_color_gold = ckd_combined[ckd_combined['Color'] == 'gold']
fig, ax = plt.subplots(figsize=(4,3))
ax.scatter(glucose_color_darkblue['White Blood Cell Count'],
glucose_color_darkblue['Glucose'],
label='Color=darkblue',
color='darkblue', s=10)
ax.scatter(glucose_color_gold['White Blood Cell Count'],
glucose_color_gold['Glucose'],
label='Color=gold',
color='gold', s=10)
x_label = 'White Blood Cell Count'
y_label = 'Glucose'
y_vals = ax.get_yticks()
plt.ylabel(y_label)
ax.legend(bbox_to_anchor=(1.04,1), loc="upper left")
plt.xlabel(x_label)
plt.show()
Earlier, we said that we expect to get some classifications wrong, because there’s some intermingling of blue and gold points in the lower-left.
But what about the points in the training set, that is, the points already on the scatter? Will we ever mis-classify them?
The answer is no. Remember that 1-nearest neighbor classification looks for the point in the training set that is nearest to the point being classified. Well, if the point being classified is already in the training set, then its nearest neighbor in the training set is itself! And therefore it will be classified as its own color, which will be correct because each point in the training set is already correctly colored.
In other words, if we use our training set to “test” our 1-nearest neighbor classifier, the classifier will pass the test 100% of the time.
Mission accomplished. What a great classifier!
No, not so much. A new point in the lower-left might easily be mis-classified, as we noted earlier. “100% accuracy” was a nice dream while it lasted.
The lesson of this example is not to use the training set to test a classifier that is based on it.
14.3.2. Generating a Test Set#
In earlier chapters, we saw that random sampling could be used to estimate the proportion of individuals in a population that met some criterion. Unfortunately, we have just seen that the training set is not like a random sample from the population of all patients, in one important respect: Our classifier guesses correctly for a higher proportion of individuals in the training set than it does for individuals in the population.
When we computed confidence intervals for numerical parameters, we wanted to have many new random samples from a population, but we only had access to a single sample. We solved that problem by taking bootstrap resamples from our sample.
We will use an analogous idea to test our classifier. We will create two samples out of the original training set, use one of the samples as our training set, and the other one for testing.
So we will have three groups of individuals:
a training set on which we can do any amount of exploration to build our classifier;
a separate testing set on which to try out our classifier and see what fraction of times it classifies correctly;
the underlying population of individuals for whom we don’t know the true classes; the hope is that our classifier will succeed about as well for these individuals as it did for our testing set.
How to generate the training and testing sets? You’ve guessed it – we’ll select at random.
There are 158 individuals in ckd. Let’s use a random half of them for training and the other half for testing. To do this, we’ll shuffle all the rows, take the first 79 as the training set, and the remaining 79 for testing.
We split before converting to standard units. The mean and SD used for scaling come from the training set only, and the same values are applied to the test set. If we scaled the whole data set first, the test patients would influence the classifier through the scaling, and the test would no longer be a fair stand-in for new patients. We also fix the random seed (random_state) so that the split, and every result below, is the same each time the page is built.
shuffled_ckd = ckd.sample(len(ckd), replace=False, random_state=12)
training_raw = shuffled_ckd.iloc[:79]
testing_raw = shuffled_ckd.iloc[79:]
# Scale both sets with the training set's mean and SD.
training = standardize(training_raw[features + ['Class']], features, training_raw).merge(color_table, on='Class')
testing = standardize(testing_raw[features + ['Class']], features, training_raw).merge(color_table, on='Class')
training.head()
| Hemoglobin | Glucose | White Blood Cell Count | Class | Color | |
|---|---|---|---|---|---|
| 0 | 0.060053 | -0.011128 | -0.341363 | 0 | gold |
| 1 | 1.272064 | -0.625135 | -0.820080 | 0 | gold |
| 2 | 0.821888 | -0.708863 | 0.169268 | 0 | gold |
| 3 | 1.133548 | -0.806546 | -0.820080 | 0 | gold |
| 4 | -1.879166 | -0.052993 | 0.328840 | 1 | darkblue |
Now let’s construct our classifier based on the points in the training sample:
training_darkblue = training[training['Color'] == 'darkblue']
training_gold = training[training['Color'] == 'gold']
fig, ax = plt.subplots(figsize=(4,3))
ax.scatter(training_darkblue['White Blood Cell Count'],
training_darkblue['Glucose'],
label='Color=darkblue',
color='darkblue', s=10)
ax.scatter(training_gold['White Blood Cell Count'],
training_gold['Glucose'],
label='Color=gold',
color='gold', s=10)
x_label = 'White Blood Cell Count'
y_label = 'Glucose'
y_vals = ax.get_yticks()
plt.ylabel(y_label)
ax.legend(bbox_to_anchor=(1.04,1), loc="upper left")
plt.xlabel(x_label)
plt.xlim(-2, 6)
plt.ylim(-2, 6);
plt.show()
# training.scatter('White Blood Cell Count', 'Glucose', group='Color')
# plt.xlim(-2, 6)
# plt.ylim(-2, 6);
We get the following classification regions and decision boundary:
x_array = np.array([])
y_array = np.array([])
for x in np.arange(-2, 6.1, 0.25):
for y in np.arange(-2, 6.1, 0.25):
x_array = np.append(x_array, x)
y_array = np.append(y_array, y)
test_grid = pd.DataFrame({
'Glucose': x_array,
'White Blood Cell Count': y_array
})
len(test_grid)
1089
len(c_training)
1089
test_grid['Class'] = c_training.astype(int)
test_grid['Class'] = test_grid['Class'].astype(str)
test_grid
| Glucose | White Blood Cell Count | Class | |
|---|---|---|---|
| 0 | -2.0 | -2.00 | 1 |
| 1 | -2.0 | -1.75 | 1 |
| 2 | -2.0 | -1.50 | 1 |
| 3 | -2.0 | -1.25 | 1 |
| 4 | -2.0 | -1.00 | 0 |
| ... | ... | ... | ... |
| 1084 | 6.0 | 5.00 | 1 |
| 1085 | 6.0 | 5.25 | 1 |
| 1086 | 6.0 | 5.50 | 1 |
| 1087 | 6.0 | 5.75 | 1 |
| 1088 | 6.0 | 6.00 | 1 |
1089 rows × 3 columns
test_grid_combined = test_grid[['Glucose', 'White Blood Cell Count', 'Class']].merge(color_table, on='Class')
test_grid_combined
| Glucose | White Blood Cell Count | Class | Color | |
|---|---|---|---|---|
| 0 | -2.0 | -2.00 | 1 | darkblue |
| 1 | -2.0 | -1.75 | 1 | darkblue |
| 2 | -2.0 | -1.50 | 1 | darkblue |
| 3 | -2.0 | -1.25 | 1 | darkblue |
| 4 | -2.0 | -1.00 | 0 | gold |
| ... | ... | ... | ... | ... |
| 1084 | 6.0 | 5.00 | 1 | darkblue |
| 1085 | 6.0 | 5.25 | 1 | darkblue |
| 1086 | 6.0 | 5.50 | 1 | darkblue |
| 1087 | 6.0 | 5.75 | 1 | darkblue |
| 1088 | 6.0 | 6.00 | 1 | darkblue |
1089 rows × 4 columns
training_darkblue = test_grid_combined[test_grid_combined['Color'] == 'darkblue']
training_gold = test_grid_combined[test_grid_combined['Color'] == 'gold']
fig, ax = plt.subplots(figsize=(4,4))
ax.scatter(training_darkblue['White Blood Cell Count'], training_darkblue['Glucose'],
label='Color=darkblue', alpha=0.4, color='darkblue', s=10)
ax.scatter(training_gold['White Blood Cell Count'], training_gold['Glucose'],
label='Color=gold', alpha=0.4, s=10, color='gold')
x_label = 'White Blood Cell Count'
y_label = 'Glucose'
y_vals = ax.get_yticks()
plt.ylabel(y_label)
ax.legend(bbox_to_anchor=(1.04,1), loc="upper left")
plt.xlabel(x_label)
plt.xlim(-2, 6)
plt.ylim(-2, 6);
plt.show()
Place the test data on this graph and you can see at once that while the classifier got almost all the points right, there are some mistakes. For example, some blue points of the test set fall in the gold region of the classifier.
import pandas as pd
import matplotlib.pyplot as plt
# 1. Add predictions to grid
test_grid['Class'] = c_training
test_grid['Class'] = test_grid['Class'].astype(int)
color_table['Class'] = color_table['Class'].astype(int)
# 2. Merge with color table on Class to attach colors
test_grid = pd.merge(test_grid, color_table, on='Class', how='left')
# 3. Scatter background grid (colored prediction regions)
fig, ax = plt.subplots(figsize=(4,4))
for color, group in test_grid.groupby("Color"):
ax.scatter(group['White Blood Cell Count'], group['Glucose'],
color=color, alpha=0.4, s=10, label='Color='+color)
# 4. Plot actual testing points
ax.scatter(testing['White Blood Cell Count'], testing['Glucose'],
c=testing['Color'], edgecolors='k', s=10)
# Formatting
ax.legend(bbox_to_anchor=(1.04,1), loc="upper left")
plt.xlabel("White Blood Cell Count")
plt.ylabel("Glucose")
plt.xlim(-2, 6)
plt.ylim(-2, 6)
plt.show()
Some errors notwithstanding, it looks like the classifier does fairly well on the test set. Assuming that the original sample was drawn randomly from the underlying population, the hope is that the classifier will perform with similar accuracy on the overall population, since the test set was chosen randomly from the original sample.
14.3.3. Classifier Accuracy#
14.3.4. Measuring the Accuracy of Our Wine Classifier#
OK, so let’s apply the hold-out method to evaluate the effectiveness of the \(k\)-nearest neighbor classifier for identifying wines. The data set has 178 wines, so we’ll randomly permute the data set and put 89 of them in the training set and the remaining 89 in the test set.
shuffled_wine = wine.sample(len(wine), replace=False, random_state=12)
training_set = shuffled_wine.take(np.arange(89))
training_set.head()
| Class | Alcohol | Malic Acid | Ash | Alcalinity of Ash | Magnesium | Total Phenols | Flavanoids | Nonflavanoid phenols | Proanthocyanins | Color Intensity | Hue | OD280/OD315 of diulted wines | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 160 | 3 | 12.36 | 3.83 | 2.38 | 21.0 | 88 | 2.30 | 0.92 | 0.50 | 1.04 | 7.65 | 0.56 | 1.58 | 520 |
| 156 | 3 | 13.84 | 4.12 | 2.38 | 19.5 | 89 | 1.80 | 0.83 | 0.48 | 1.56 | 9.01 | 0.57 | 1.64 | 480 |
| 78 | 2 | 12.33 | 0.99 | 1.95 | 14.8 | 136 | 1.90 | 1.85 | 0.35 | 2.76 | 3.40 | 1.06 | 2.31 | 750 |
| 45 | 1 | 14.21 | 4.04 | 2.44 | 18.9 | 111 | 2.85 | 2.65 | 0.30 | 1.25 | 5.24 | 0.87 | 3.33 | 1080 |
| 112 | 2 | 11.76 | 2.68 | 2.92 | 20.0 | 103 | 1.75 | 2.03 | 0.60 | 1.05 | 3.80 | 1.23 | 2.50 | 607 |
test_set = shuffled_wine.take(np.arange(89, 178))
test_set.head()
| Class | Alcohol | Malic Acid | Ash | Alcalinity of Ash | Magnesium | Total Phenols | Flavanoids | Nonflavanoid phenols | Proanthocyanins | Color Intensity | Hue | OD280/OD315 of diulted wines | Proline | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 1 | 13.16 | 2.36 | 2.67 | 18.6 | 101 | 2.80 | 3.24 | 0.30 | 2.81 | 5.68 | 1.03 | 3.17 | 1185 |
| 28 | 1 | 13.87 | 1.90 | 2.80 | 19.4 | 107 | 2.95 | 2.97 | 0.37 | 1.76 | 4.50 | 1.25 | 3.40 | 915 |
| 148 | 3 | 13.32 | 3.24 | 2.38 | 21.5 | 92 | 1.93 | 0.76 | 0.45 | 1.25 | 8.42 | 0.55 | 1.62 | 650 |
| 132 | 3 | 12.81 | 2.31 | 2.40 | 24.0 | 98 | 1.15 | 1.09 | 0.27 | 0.83 | 5.70 | 0.66 | 1.36 | 560 |
| 9 | 1 | 13.86 | 1.35 | 2.27 | 16.0 | 98 | 2.98 | 3.15 | 0.22 | 1.85 | 7.22 | 1.01 | 3.55 | 1045 |
We’ll train the classifier using the 89 wines in the training set, and evaluate how well it performs on the test set. To make our lives easier, we’ll write a function to evaluate a classifier on every wine in the test set:
def count_zero(array):
"""Counts the number of 0's in an array"""
return len(array) - np.count_nonzero(array)
def count_equal(array1, array2):
"""Takes two numerical arrays of equal length
and counts the indices where the two are equal"""
return count_zero(array1 - array2)
def evaluate_accuracy(training, test, k):
test_attributes = test.drop(columns=['Class'])
def classify_testrow(row):
return classify(training, row, k)
c = test_attributes.apply(classify_testrow, axis=1)
return count_equal(c, test['Class']) / len(test)
Now for the grand reveal – let’s see how we did. We’ll arbitrarily use \(k=5\).
evaluate_accuracy(training_set, test_set, 5)
np.float64(0.6741573033707865)
About 67% of the test wines are classified correctly. That is better than guessing among three cultivars, but the classifier is missing something. Look at the columns in training_set.head() above: Proline is in the hundreds or thousands, while most other attributes are below 10. Distance adds up squared differences, so a difference of 100 in Proline swamps every other attribute, and the classifier is effectively using one attribute.
Scaling the features. As with the kidney data, we convert each attribute to standard units. The mean and SD come from the training set, and the same values are applied to the test set.
wine_features = wine.columns.drop('Class')
training_scaled = standardize(training_set, wine_features, training_set)
test_scaled = standardize(test_set, wine_features, training_set)
evaluate_accuracy(training_scaled, test_scaled, 5)
np.float64(0.9550561797752809)
Once every attribute counts on the same scale, the same classifier, with the same training and test wines, is far more accurate: about 96% of the test wines are now classified correctly, up from 67%. Scaling is part of building a nearest-neighbor classifier, not an optional extra.
14.3.5. Breast Cancer Diagnosis#
Now I want to do an example based on diagnosing breast cancer. I was inspired by Brittany Wenger, who won the Google national science fair in 2012 as a 17-year old high school student. Here’s Brittany:
Brittany’s science fair project was to build a classification algorithm to diagnose breast cancer. She won grand prize for building an algorithm whose accuracy was almost 99%.
Let’s see how well we can do, with the ideas we’ve learned in this course.
So, let me tell you a little bit about the data set. Basically, if a woman has a lump in her breast, the doctors may want to take a biopsy to see if it is cancerous. There are several different procedures for doing that. Brittany focused on fine needle aspiration (FNA), because it is less invasive than the alternatives. The doctor gets a sample of the mass, puts it under a microscope, takes a picture, and a trained lab tech analyzes the picture to determine whether it is cancer or not. We get a picture like one of the following:
Unfortunately, distinguishing between benign vs malignant can be tricky. So, researchers have studied the use of machine learning to help with this task. The idea is that we’ll ask the lab tech to analyze the image and compute various attributes: things like the typical size of a cell, how much variation there is among the cell sizes, and so on. Then, we’ll try to use this information to predict (classify) whether the sample is malignant or not. We have a training set of past samples from women where the correct diagnosis is known, and we’ll hope that our machine learning algorithm can use those to learn how to predict the diagnosis for future samples.
We end up with the following data set. For the “Class” column, 1 means malignant (cancer); 0 means benign (not cancer).
patients = pd.read_csv(path_data + 'breast-cancer.csv').drop(columns='ID')
patients
| Clump Thickness | Uniformity of Cell Size | Uniformity of Cell Shape | Marginal Adhesion | Single Epithelial Cell Size | Bare Nuclei | Bland Chromatin | Normal Nucleoli | Mitoses | Class | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 5 | 1 | 1 | 1 | 2 | 1 | 3 | 1 | 1 | 0 |
| 1 | 5 | 4 | 4 | 5 | 7 | 10 | 3 | 2 | 1 | 0 |
| 2 | 3 | 1 | 1 | 1 | 2 | 2 | 3 | 1 | 1 | 0 |
| 3 | 6 | 8 | 8 | 1 | 3 | 4 | 3 | 7 | 1 | 0 |
| 4 | 4 | 1 | 1 | 3 | 2 | 1 | 3 | 1 | 1 | 0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 678 | 3 | 1 | 1 | 1 | 3 | 2 | 1 | 1 | 1 | 0 |
| 679 | 2 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 0 |
| 680 | 5 | 10 | 10 | 3 | 7 | 3 | 8 | 10 | 2 | 1 |
| 681 | 4 | 8 | 6 | 4 | 3 | 4 | 10 | 6 | 1 | 1 |
| 682 | 4 | 8 | 8 | 5 | 4 | 5 | 10 | 4 | 1 | 1 |
683 rows × 10 columns
So we have 9 different attributes. I don’t know how to make a 9-dimensional scatterplot of all of them, so I’m going to pick two and plot them:
color_table = pd.DataFrame({
'Class': [1, 0],
'Color': ['darkblue', 'gold']
})
patients_with_colors = patients.merge(color_table, on='Class')
fig, ax = plt.subplots(figsize=(5, 4))
for color, group in patients_with_colors.groupby('Color'):
ax.scatter(group['Bland Chromatin'], group['Single Epithelial Cell Size'],
color=color, label=color, alpha=0.7, s=15)
ax.set_xlabel('Bland Chromatin')
ax.set_ylabel('Single Epithelial Cell Size')
ax.legend()
plt.show()
Oops. That plot is utterly misleading, because there are a bunch of points that have identical values for both the x- and y-coordinates. To make it easier to see all the data points, I’m going to add a little bit of random jitter to the x- and y-values. Here’s how that looks:
For instance, you can see there are lots of samples with chromatin = 2 and epithelial cell size = 2; all non-cancerous.
Keep in mind that the jittering is just for visualization purposes, to make it easier to get a feeling for the data. We’re ready to work with the data now, and we’ll use the original (unjittered) data.
First we’ll create a training set and a test set. The data set has 683 patients, so we’ll randomly permute the data set and put 342 of them in the training set and the remaining 341 in the test set.
shuffled_patients = patients.sample(frac=1, random_state=12).reset_index(drop=True)
training_set = shuffled_patients.iloc[:342]
test_set = shuffled_patients.iloc[342:]
Let’s stick with 5 nearest neighbors, and see how well our classifier does.
evaluate_accuracy(training_set, test_set, 5)
np.float64(0.9736070381231672)
Almost 98% accuracy. Not bad! Once again, pretty darn good for such a simple technique.
As a footnote, you might have noticed that Brittany Wenger did even better. What techniques did she use? One key innovation is that she incorporated a confidence score into her results: her algorithm had a way to determine when it was not able to make a confident prediction, and for those patients, it didn’t even try to predict their diagnosis. Her algorithm was 99% accurate on the patients where it made a prediction – so that extension seemed to help quite a bit.
14.3.6. Section Practice#
Use this short exercise to check the main workflow from this section.
### Exercise: Practice Training, Testing, and Accuracy
# A 1-nearest neighbor classifier was checked twice: on its own training set
# and on a separate test set. Its predictions and the actual labels are below.
# 1. Compute the accuracy (proportion of correct predictions) on each set.
# 2. Print both. Which one is the honest estimate for new customers?
# Print exactly:
# Training accuracy: 1.0
# Test accuracy: 0.75
train_predicted = ['yes', 'no', 'no', 'yes', 'no', 'yes']
train_actual = ['yes', 'no', 'no', 'yes', 'no', 'yes']
test_predicted = ['yes', 'no', 'yes', 'no', 'no', 'yes', 'no', 'yes']
test_actual = ['yes', 'no', 'no', 'no', 'yes', 'yes', 'no', 'yes']
### Your code starts here
### Your code ends here
Training accuracy: 1.0
Test accuracy: 0.75