Performance Binominal Classification

Synopsis

This Operator is used to statistically evaluate the strengths and weaknesses of a binary classification, after a trained model has been applied to labelled data.

Description

A binary classification makes predictions where the outcome has two possible values: call them positive and negative. Moreover, the prediction for each Example may be right or wrong, leading to a 2x2 confusion matrix with 4 entries:

TP - the number of "true positives", positive Examples that have been correctly identified
FP - the number of "false positives", negative Examples that have been incorrectly identified
FN - the number of "false negatives", positive Examples that have been incorrectly identified
TN - the number of "true negatives", negative Examples that have been correctly identified

In the parameter section, numerous performance criteria are described, any of which can be calculated in terms of the above variables.

If the model has a probabilistic scoring system where scores above a certain threshold are identified as positive, then the elements of the confusion matrix will depend on the threshold. To create an ROC graph and calculate the area under the curve (AUC), the threshold is varied and a point (x, y) is plotted for each threshold value:

y-axis - true positive rate = (True positive predictions)/(Number of positive Examples) = TP / (TP + FN)
x-axis - false positive rate = (False positive predictions)/(Number of negative Examples) = FP / (FP + TN)

Differentiation

There are numerous performance Operators, and you should choose the one that is best suited to your problem.

Performance (Classification)

Choose this Operator when the label is nominal and it has more than two values.

Input

labeled data

这个输入端口预计ExampleSet的标签。使sure the ExampleSet has both a label Attribute and a prediction Attribute, and that the label is of type binominal.

performance

This input port expects a performance vector. You need to connect a performance vector to the input if you want to do multi-objective optimization.

Output

performance

This output port delivers a performance vector -- a list of performance criterion values based on the label and prediction Attributes of the input ExampleSet.

In the output, the performance criterion values from the input (if any) are combined with the values from this Operator; in case of overlap, the values from the input are overwritten.

example set

The ExampleSet that was given as input is passed through without changes.

Parameters

Main criterion

The main criterion is used when performance vectors are compared, e.g., parameter Optimization or Attribute selection. If not selected, the main criterion is the first criterion in the output performance vector.

If performance vectors are not compared, the main criterion is ignored.

Manually set positive class

Check this box to use thepositive classparameter to manually specify the positive class. Otherwise the positive class is derived from the label's internal mapping. It is recommended to manually set the desired positive class.

Positive class

Use this parameter to manually set the positive class.

In rare cases the suggested values in the drop down menu may not match the actual label's values. In this case the correct positive class can be specified by typing its name manually instead of selecting it from the drop down menu.

Accuracy

accuracy = (Correct predictions)/(Number of Examples) = (TP + TN) / (TP + FP + FN + TN)

Classification error

classification error = (Incorrect predictions)/(Number of Examples) = (FP + FN) / (TP + FP + FN + TN)

Kappa

Cohen's kappa = (po - pe)/(1 - pe)

where:

po = observed accuracy = (TP + TN) / (TP + FP + FN + TN)

pe = expected accuracy =[(TP + FP)(TP + FN) + (FN + TN)(FP + TN)]/[(TP + FP + FN + TN)^2]

Auc (optimistic)

When the ROC graph is plotted, before calculating the area under the curve (AUC), the predictions are sorted by score, from highest to lowest, and the graph is plotted Example by Example. If two or more Examples have the same score, the ordering is not well-defined; in this case, the optimistic version of AUC plots the positive Examples before plotting the negative Examples.

Auc

When the ROC graph is plotted, before calculating the area under the curve (AUC), the predictions are sorted by score, from highest to lowest, and the graph is plotted Example by Example. If two or more Examples have the same score, the ordering is not well-defined. The normal version of AUC calculates the area by taking the average of AUC (optimistic) and AUC (pessimistic).

Auc (pessimistic)

When the ROC graph is plotted, before calculating the area under the curve (AUC), the predictions are sorted by score, from highest to lowest, and the graph is plotted Example by Example. If two or more Examples have the same score, the ordering is not well-defined; in this case, the pessimistic version of AUC plots the negative Examples before plotting the positive Examples.

Precision

precision = (True positive predictions)/(All positive predictions) = TP / (TP + FP)

Recall

recall = (True positive predictions)/(Number of positive Examples) = TP / (TP + FN)

Lift

lift is the ratio of two quantities, representing the improvement over random sampling:

The probability of choosing a positive Example from the group of all positive predictions: TP / (TP + FP)
The probability of choosing a positive Example from the group of all Examples: (TP + FN) / (TP + FP + FN + TN)

lift =[TP / (TP + FP)]/[(TP + FN) / (TP + FP + FN + TN)]

Fallout

fallout = (False positive predictions)/(Number of negative Examples) = FP / (FP + TN)

F measure

F1 = 2 (precision * recall) / (precision + recall) = 2TP / (2TP + FP + FN)

False positive

假阳性的数量预测:《外交政策》

False negative

The number of false negative predictions: FN

True positive

The number of true positive predictions: TP

True negative

The number of true negative predictions: TN

Sensitivity

sensitivity = recall = (True positive predictions)/(Number of positive Examples) = TP / (TP + FN)

Specificity

specificity = (True negative predictions)/(Number of negative Examples) = TN / (TN + FP)

Youden

Sometimes called informedness or DeltaP'.

J = sensitivity + specificity - 1

Positive predictive value

PPV = precision = (True positive predictions)/(All positive predictions) = TP / (TP + FP)

Negative predictive value

NPV = (True negative predictions)/(All negative predictions) = TN / (TN + FN)

Psep

Sometimes called markedness or DeltaP.

psep = PPV + NPV - 1

Skip undefined labels

When this parameter is true, Examples not belonging to a defined class are ignored.

Comparator class

The fully qualified classname of the PerformanceComparator implementation is specified here.

Use example weights

This parameter has no effect if no Attribute has the weight role.