What is TunedThresholdClassifierCV?
TunedThresholdClassifierCV
is a utility in scikit-learn that helps in finding the optimal threshold for decision-making in binary classification. It uses cross-validation to evaluate different thresholds and select the one that maximizes a specified metric, such as F1-score, precision, recall, or any other custom metric.
Installation
Before using TunedThresholdClassifierCV
, ensure you have scikit-learn installed. You can install it using pip:
pip install scikit-learn
Step 1: Import Libraries
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, precision_recall_curve, classification_report
Step 2: Create Synthetic Dataset
- make_classification: Generates a synthetic dataset with 1000 samples and 20 features.
- train_test_split: Splits the dataset into training (80%) and testing (20%) sets.
# Create a synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
Step 3: Define and Tune the Model
- RandomForestClassifier: Initializes a random forest classifier.
- param_grid: Defines the hyperparameters to tune (number of trees and maximum depth).
- GridSearchCV: Performs a grid search with cross-validation (5 folds) to find the best hyperparameters based on the ROC AUC score.
- search.fit: Fits the grid search on the training data.
# Define a RandomForestClassifier
model = RandomForestClassifier()
# Define the parameter grid for GridSearchCV
param_grid = {
'n_estimators': [50, 100, 200],
'max_depth': [None, 10, 20, 30]
}
# Perform GridSearchCV to find the best model parameters
search = GridSearchCV(model, param_grid, scoring='roc_auc', cv=5)
search.fit(X_train, y_train)
Step 4: Evaluate the Best Model
- best_model: Retrieves the best model from the grid search.
- predict_proba: Predicts the probabilities of the positive class for the test set.
- roc_auc_score: Calculates the ROC AUC score for the predicted probabilities at the default threshold (0.5).
- print: Prints the ROC AUC score.
# Get the best model
best_model = search.best_estimator_
# Predict probabilities
y_probs = best_model.predict_proba(X_test)[:, 1]
# Evaluate model performance at the default threshold (0.5)
roc_auc = roc_auc_score(y_test, y_probs)
print(f"ROC AUC at default threshold: {roc_auc}")
Output:
ROC AUC at default threshold: 0.9267410310521557
Step 5:Find the Best Threshold
# Find the best threshold using Precision-Recall curve
precision, recall, thresholds = precision_recall_curve(y_test, y_probs)
fscore = (2 * precision * recall) / (precision + recall)
best_threshold = thresholds[np.argmax(fscore)]
print(f"Best threshold: {best_threshold}")
Output:
Best threshold: 0.4942682076551753
6. Apply the Best Threshold
y_pred_best_threshold: Converts probabilities to binary predictions based on the best threshold.
# Apply the best threshold
y_pred_best_threshold = (y_probs >= best_threshold).astype(int)
7. Evaluate Performance at the Best Threshold
- roc_auc_score: Calculates the ROC AUC score for the predictions based on the best threshold.
- print: Prints the ROC AUC score at the best threshold.
- classification_report: Generates and prints a detailed classification report including precision, recall, and F1-score for each class.
# Evaluate model performance at the best threshold
roc_auc_best_threshold = roc_auc_score(y_test, y_pred_best_threshold)
print(f"ROC AUC at best threshold: {roc_auc_best_threshold}")
# Print classification report
print(classification_report(y_test, y_pred_best_threshold))
Output:
ROC AUC at best threshold: 0.8976484775399458
precision recall f1-score support
0 0.85 0.94 0.89 93
1 0.94 0.86 0.90 107
accuracy 0.90 200
macro avg 0.90 0.90 0.89 200
weighted avg 0.90 0.90 0.90 200
The ROC AUC at the best threshold is 0.8976, indicating a high ability to distinguish between positive and negative classes. For class 0, the precision is 0.85, recall is 0.94, and F1-score is 0.89. For class 1, the precision is 0.94, recall is 0.86, and F1-score is 0.90, demonstrating the model’s strong performance in correctly identifying both classes.
The accuracy of the model is 90%, meaning that 90% of all instances were correctly classified. The macro average, which averages metrics for each class independently, shows a balanced performance with precision, recall, and F1-score all around 0.90, indicating that the model performs well across both classes without significant bias towards either class.
How to use scikit-learn’s TunedThresholdClassifierCV for Threshold Optimization?
Threshold optimization is crucial in many machine learning tasks, particularly in binary classification, where the decision boundary needs fine-tuning to balance precision and recall. Scikit-learn’s TunedThresholdClassifierCV
provides a streamlined way to optimize thresholds, leveraging cross-validation to find the best threshold that improves model performance. In this article, we will discuss
Contact Us