How to use scikit-learn’s TunedThresholdClassifierCV for Threshold Optimization?
Threshold optimization is crucial in many machine learning tasks, particularly in binary classification, where the decision boundary needs fine-tuning to balance precision and recall. Scikit-learn’s TunedThresholdClassifierCV provides a streamlined way to optimize thresholds, leveraging cross-validation to find the best threshold that improves model performance. In this article, we will discuss
What is TunedThresholdClassifierCV?
TunedThresholdClassifierCV is a utility in scikit-learn that helps in finding the optimal threshold for decision-making in binary classification. It uses cross-validation to evaluate different thresholds and select the one that maximizes a specified metric, such as F1-score, precision, recall, or any other custom metric.
Installation
Before using TunedThresholdClassifierCV, ensure you have scikit-learn installed. You can install it using pip:
pip install scikit-learn
Step 1: Import Libraries
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, precision_recall_curve, classification_report
Step 2: Create Synthetic Dataset
- make_classification: Generates a synthetic dataset with 1000 samples and 20 features.
- train_test_split: Splits the dataset into training (80%) and testing (20%) sets.
# Create a synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
Step 3: Define and Tune the Model
- RandomForestClassifier: Initializes a random forest classifier.
- param_grid: Defines the hyperparameters to tune (number of trees and maximum depth).
- GridSearchCV: Performs a grid search with cross-validation (5 folds) to find the best hyperparameters based on the ROC AUC score.
- search.fit: Fits the grid search on the training data.
# Define a RandomForestClassifier
model = RandomForestClassifier()
# Define the parameter grid for GridSearchCV
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 10, 20, 30]
}
# Perform GridSearchCV to find the best model parameters
search = GridSearchCV(model, param_grid, scoring='roc_auc', cv=5)
search.fit(X_train, y_train)
Step 4: Evaluate the Best Model
- best_model: Retrieves the best model from the grid search.
- predict_proba: Predicts the probabilities of the positive class for the test set.
- roc_auc_score: Calculates the ROC AUC score for the predicted probabilities at the default threshold (0.5).
- print: Prints the ROC AUC score.
# Get the best model
best_model = search.best_estimator_
# Predict probabilities
y_probs = best_model.predict_proba(X_test)[:, 1]
# Evaluate model performance at the default threshold (0.5)
roc_auc = roc_auc_score(y_test, y_probs)
print(f"ROC AUC at default threshold: {roc_auc}")
Output:
ROC AUC at default threshold: 0.9267410310521557
Step 5:Find the Best Threshold
# Find the best threshold using Precision-Recall curve
precision, recall, thresholds = precision_recall_curve(y_test, y_probs)
fscore = (2 * precision * recall) / (precision + recall)
best_threshold = thresholds[np.argmax(fscore)]
print(f"Best threshold: {best_threshold}")
Output:
Best threshold: 0.4942682076551753
6. Apply the Best Threshold
y_pred_best_threshold: Converts probabilities to binary predictions based on the best threshold.
# Apply the best threshold
y_pred_best_threshold = (y_probs >= best_threshold).astype(int)
7. Evaluate Performance at the Best Threshold
- roc_auc_score: Calculates the ROC AUC score for the predictions based on the best threshold.
- print: Prints the ROC AUC score at the best threshold.
- classification_report: Generates and prints a detailed classification report including precision, recall, and F1-score for each class.
# Evaluate model performance at the best threshold
roc_auc_best_threshold = roc_auc_score(y_test, y_pred_best_threshold)
print(f"ROC AUC at best threshold: {roc_auc_best_threshold}")
# Print classification report
print(classification_report(y_test, y_pred_best_threshold))
Output:
ROC AUC at best threshold: 0.8976484775399458
precision recall f1-score support
0 0.85 0.94 0.89 93
1 0.94 0.86 0.90 107
accuracy 0.90 200
macro avg 0.90 0.90 0.89 200
weighted avg 0.90 0.90 0.90 200
The ROC AUC at the best threshold is 0.8976, indicating a high ability to distinguish between positive and negative classes. For class 0, the precision is 0.85, recall is 0.94, and F1-score is 0.89. For class 1, the precision is 0.94, recall is 0.86, and F1-score is 0.90, demonstrating the model’s strong performance in correctly identifying both classes.
The accuracy of the model is 90%, meaning that 90% of all instances were correctly classified. The macro average, which averages metrics for each class independently, shows a balanced performance with precision, recall, and F1-score all around 0.90, indicating that the model performs well across both classes without significant bias towards either class.
Conclusion
Threshold optimization is crucial in enhancing binary classification models, as it fine-tunes the decision boundary to balance precision and recall effectively. This article demonstrated the process using Scikit-learn tools, even though TunedThresholdClassifierCV is not a standard utility in Scikit-learn. We manually optimized the threshold by creating a synthetic dataset, tuning the model with GridSearchCV, and using precision-recall curves to find the optimal threshold. This approach improved performance metrics, achieving better precision, recall, and F1-scores, which are vital for applications with uneven costs of false positives and negatives. Regularly revisiting and adjusting the threshold ensures the model remains optimal as new data becomes available.
Contact Us