after building model:

• chose the best model
• estimate model’s performance
• make predictions for unseen documents

Metrics

• with labelled data evaluation works by comparing the labels we gave to the labels the model predicts.
  • the closer the predicted labels are to the true labels, the better the model
  • metrics quantify diff aspects of that “closeness”

Accuracy

• frequently used → easy to understand + explain
• proportion of predictions which are correct

accuracy = correct predictions/total predictions

import numpyas np

# Let's examiney_pred= np.array([0, 0, 0, 0, 0, 1, 1, 0, 0, 0])
y_true= np.array([0, 0, 0, 0, 1, 1, 0, 0, 0, 0])

def calculate_accuracy(y_true, y_pred):
    total_predictions= y_true.size
    correct_predictions= (y_pred==y_true).sum()
    accuracy= correct_predictions/ total_predictions
    print(f"{correct_predictions} correct predictions out of {total_predictions} total predictions.\\nAccuracy = {accuracy:.0%}")

calculate_accuracy(y_true, y_pred)

Accuracy only says a little because under very common conditions → accuracy isn’t accurate

we need specific info about how the model performs. we need data for what kind of mistakes the model makes most frequently.

precision + recall

predictions have 2 common types of errors

• false positives → a sample is negative and it is predicted to be positive
• false negatives → where a sample is positive and it is predicted to be negative

2 types of correct predictions

• true positives