Data Mining_ Concepts and Techniques - Jiawei Han [225]
Figure 8.13 Evaluation measures. Note that some measures are known by more than one name. TP, TN, FP, P, N refer to the number of true positive, true negative, false positive, positive, and negative samples, respectively (see text).
Using training data to derive a classifier and then estimate the accuracy of the resulting learned model can result in misleading over-optimistic estimates due to over-specialization of the learning algorithm to the data. (We will say more on this in a moment!) Instead, it is better to measure the classifier's accuracy on a test set consisting of class-labeled tuples that were not used to train the model.
Before we discuss the various measures, we need to become comfortable with some terminology. Recall that we can talk in terms of positive tuples (tuples of the main class of interest) and negative tuples (all other tuples). 6 Given two classes, for example, the positive tuples may be buys_computer = yes while the negative tuples are buys_computer = no. Suppose we use our classifier on a test set of labeled tuples. P is the number of positive tuples and N is the number of negative tuples. For each tuple, we compare the classifier's class label prediction with the tuple's known class label.
6In the machine learning and pattern recognition literature, these are referred to as positive samples and negative samples, respectively.
There are four additional terms we need to know that are the “building blocks” used in computing many evaluation measures. Understanding them will make it easy to grasp the meaning of the various measures.
■ True positives (TP): These refer to the positive tuples that were correctly labeled by the classifier. Let TP be the number of true positives.
■ True negatives (TN): These are the negative tuples that were correctly labeled by the classifier. Let TN be the number of true negatives.
■ False positives (FP): These are the negative tuples that were incorrectly labeled as positive (e.g., tuples of class buys_computer = no for which the classifier predicted buys_computer = yes). Let FP be the number of false positives.
■ False negatives (FN): These are the positive tuples that were mislabeled as negative (e.g., tuples of class buys_computer = yes for which the classifier predicted buys_computer = no). Let FN be the number of false negatives.
These terms are summarized in the confusion matrix of Figure 8.14.
Figure 8.14 Confusion matrix, shown with totals for positive and negative tuples.
The confusion matrix is a useful tool for analyzing how well your classifier can recognize tuples of different classes. TP and TN tell us when the classifier is getting things right, while FP and FN tell us when the classifier is getting things wrong (i.e., mislabeling). Given m classes (where ), a confusion matrix is a table of at least size m by m. An entry, in the first m rows and m columns indicates the number of tuples of class i that were labeled by the classifier as class j. For a classifier to have good accuracy, ideally most of the tuples would be represented along the diagonal of the confusion matrix, from entry to entry , with the rest of the entries being zero or close to zero. That is, ideally, FP and FN are around zero.
The table may have additional rows or columns to provide totals. For example, in the confusion matrix of Figure 8.14, P and N are shown. In addition, is the number of tuples that were labeled as positive and is the number of tuples that were labeled as negative . The total number of tuples is , or , or . Note that although the confusion matrix shown is for a binary classification problem, confusion matrices can be easily drawn for multiple classes in a similar manner.
Now let's look at the evaluation measures, starting with accuracy. The accuracy of a classifier on a given test set is the percentage of test set tuples that are correctly classified by the classifier. That is,
(8.21)
In the pattern recognition