Introduction
So, you’ve built a machine learning model and it gives you 95% accuracy. Sounds amazing, right? But what if your data is highly imbalanced—say, 95% of your samples belong to one class? In that case, your model may be predicting only the majority class and still achieving 95% accuracy without learning anything meaningful.
Welcome to the world of model evaluation metrics, where accuracy is just the tip of the iceberg.
In this blog, we’ll explore essential model evaluation metrics—classification and regression, including real-world analogies, when to use what, and code snippets to solidify your understanding.
Why Metrics Matter
A machine learning model is only as good as its ability to generalize to unseen data. Evaluation metrics help answer key questions:
How well is my model performing?
Is it overfitting or underfitting?
Should I improve the data, the model, or the features?
Different problems need different metrics. Let’s break it down.
Classification Metrics
1. Accuracy
Accuracy measures how often the classifier is correct overall. It is the ratio of correctly predicted observations (both true positives and true negatives) to the total number of observations.
Formula:
Accuracy =(TP+TN)/(TP+TN+FP+FN)
Intuition:
Imagine you have 100 emails, and your model classifies 90 correctly — regardless of class. Then your accuracy is 90%.
Limitation: In imbalanced datasets, accuracy can be high even if the model completely ignores the minority class.
2. Confusion Matrix
A confusion matrix is a 2D matrix used to visualize the performance of a classification algorithm. It compares the actual labels with predicted labels and breaks them down into:
True Positives (TP): Correctly predicted positive class
True Negatives (TN): Correctly predicted negative class
False Positives (FP): Predicted positive, but it’s actually negative (Type I error)
False Negatives (FN): Predicted negative, but it’s actually positive (Type II error)
Why it matters:
It helps you go beyond accuracy to understand exactly where your model is making mistakes.
3. Precision
Precision is the proportion of predicted positives that are truly positive. It tells us how many of the predicted positive results were actually correct.
Formula:
Precision=(TP)/(TP+FP)
Example:
If a model predicts 10 emails as spam, but only 7 are actually spam, then precision is 70%.
High precision = low false positive rate.
4. Recall (Sensitivity / True Positive Rate)
Recall measures the ability of a model to detect all relevant cases within a dataset. It is the ratio of correctly predicted positive observations to all actual positives.
Formula:
Recall=(TP)/(TP+FN)
Example:
If there are 100 spam emails and your model correctly identifies 80 of them, recall is 80%.
High recall = low false negative rate.
5. F1-Score
The F1-score is the harmonic mean of Precision and Recall. It balances both concerns into a single metric. The harmonic mean penalizes extreme values more, which makes F1 a balanced score even when Precision and Recall vary significantly.
Formula:
F1=(2×(Precision×Recall)/(Precision+Recall))
Why it matters:
Useful in situations where both false positives and false negatives are costly — especially on imbalanced datasets.
6. ROC Curve (Receiver Operating Characteristic Curve)
ROC is a graphical representation of a model’s performance across all classification thresholds.
It plots:
TPR = Recall
FPR = FP / (FP + TN)
A good model has a curve close to the top-left corner.
7. AUC (Area Under ROC Curve)
AUC represents the area under the ROC curve. It gives an aggregate measure of performance across all possible classification thresholds. A perfect model has an AUC of 1.0, while a model that randomly guesses has an AUC of 0.5.
Why it matters:
It is threshold-independent and useful for comparing multiple classifiers.
8. Log Loss (Cross-Entropy Loss)
Log Loss measures the uncertainty of the model’s predictions by penalizing false classifications and poorly calibrated probability predictions.
Formula (Binary Classification):
Log Loss=−(1/n)∑[yi log(pi)+(1−yi)log(1−pi)]
Why it matters:
It encourages well-calibrated probabilities. Lower log loss indicates better performance.
Regression Metrics
1. Mean Absolute Error (MAE)
MAE measures the average magnitude of errors in a set of predictions, without considering their direction (i.e., it doesn’t matter whether the error is positive or negative).
Formula:
MAE=(1/n)∑∣yi−yi^∣
Why it matters:
It is easy to understand and interpret. Unlike MSE, it doesn’t penalize large errors more heavily.
2. Mean Squared Error (MSE)
MSE measures the average squared difference between actual and predicted values. Squaring the error magnifies larger errors.
Formula:
MSE=(1/n)∑(yi−yi^)^^2
Why it matters:
It’s sensitive to outliers and gives a stronger signal when large errors exist.
3. Root Mean Squared Error (RMSE)
RMSE is the square root of MSE. It brings the error back to the same units as the target variable, making it easier to interpret.
Formula:
RMSE=Sqrt(MSE)=Sqrt((1/n)∑(yi−yi^)^^2)
Why it matters:
It’s one of the most commonly used metrics and more interpretable than MSE due to unit consistency.
4. R² Score (Coefficient of Determination)
R² measures the proportion of the variance in the dependent variable that is predictable from the independent variables. It compares your model to a naive model that always predicts the mean.
Formula:
R2=(1−(SSres)/(SStot))=1−(∑(yi−yi^)^^2)/(∑(yi−yˉ)^^2)
- SSres : sum of squared residuals
- SStot: total sum of squares
Why it matters:
A value closer to 1 means better model performance. A value < 0 indicates the model performs worse than the mean prediction.
Tip for Readers
✅ Always choose metrics based on:
- Problem type (classification vs regression)
- Class distribution
- Business impact of false positives/negatives