Home achine Learning XGBoost in R: How Does xgb.cv Pass the Optimal Parameters into xgb.train

XGBoost in R: How Does xgb.cv Pass the Optimal Parameters into xgb.train

Set or View the Graphics Palette in R Programming - palette() Function

Python Numpy

XGBoost is a powerful gradient-boosting algorithm known for its efficiency and effectiveness in handling structured data. When tuning hyperparameters for an XGBoost model, cross-validation (CV) is commonly used to find the optimal combination of parameters. The xgb.cv function in R performs cross-validation to identify the best parameters, but how does it pass these optimal parameters into xgb. train for model training? This article aims to explain the process and provide insights into how xgb. cv optimizes hyperparameters and passes them to xgb. train.

What is xgb. cv?

xgb. cv is a function in the xgboost package that performs k-fold cross-validation to evaluate the performance of an XGBoost model and tune its hyperparameters. It systematically trains and evaluates multiple models using different combinations of hyperparameters to find the optimal set that maximizes a specified evaluation metric.

How Does xgb.cv Work?

  • Parameter Grid: xgb. cv defines a grid of hyperparameters to explore. This grid typically includes parameters like max_depth, learning_rate, subsample, and colsample_bytree.
  • Cross-Validation: The dataset is split into k folds, and for each combination of hyperparameters in the grid:
    • A model is trained on k-1 folds.
    • The performance is evaluated on the remaining fold.
    • This process is repeated for each fold, and the average performance across folds is calculated.
  • Optimization: xgb.cv identifies the set of hyperparameters that maximizes the specified evaluation metric (e.g., accuracy, AUC, RMSE).

Passing Optimal Parameters to xgb.train

Once the optimal parameters are identified, xgb.cv passes them to xgb.train for model training. But how does this transfer of parameters occur?

Specifying Optimal Parameters

The xgb.cv function returns a list of results, including the optimal set of hyperparameters found during cross-validation. These parameters can be extracted using the $best.params attribute.

R 
library(xgboost)
library(Matrix)

data(iris)
iris$Species <- as.numeric(iris$Species) - 1
X <- as.matrix(iris[, -5])
y <- iris$Species
dtrain <- xgb.DMatrix(data = X, label = y)

Define Parameters and Perform Cross-Validation

Instead of providing a list of values directly in the params argument, define a set of parameters to test within the cross-validation loop.

R 
# Set initial parameters
params <- list(
  objective = "multi:softprob",
  eval_metric = "mlogloss",
  num_class = 3
)

# Perform cross-validation
cv_results <- xgb.cv(
  params = params,
  data = dtrain,
  nfold = 5,
  nrounds = 100,
  early_stopping_rounds = 10,
  verbose = TRUE
)

Output:

[1]    train-mlogloss:0.738994+0.006730    test-mlogloss:0.749825+0.018149 
Multiple eval metrics are present. Will use test_mlogloss for early stopping.
Will train until test_mlogloss hasn't improved in 10 rounds.

[2]    train-mlogloss:0.528526+0.009564    test-mlogloss:0.549305+0.032133 
[3]    train-mlogloss:0.391399+0.011049    test-mlogloss:0.420857+0.043757 
[4]    train-mlogloss:0.297514+0.010924    test-mlogloss:0.334201+0.053962 
[5]    train-mlogloss:0.230494+0.010313    test-mlogloss:0.275613+0.062684 
[6]    train-mlogloss:0.181601+0.010129    test-mlogloss:0.235692+0.070476 
[7]    train-mlogloss:0.144770+0.009516    test-mlogloss:0.208053+0.078296 
[8]    train-mlogloss:0.117297+0.009185    test-mlogloss:0.188952+0.084840 
[9]    train-mlogloss:0.096336+0.008522    test-mlogloss:0.176641+0.091872 
[10]    train-mlogloss:0.080486+0.007804    test-mlogloss:0.167870+0.097877

Passing Optimal Parameters to xgb.train

The optimal parameters are then passed as input to xgb.train when training the final model.

R 
# Extract the Best Number of Rounds
best_nrounds <- cv_results$best_iteration
# Train the Final Model with Optimal Parameters
final_model <- xgb.train(
  params = params,
  data = dtrain,
  nrounds = best_nrounds
)
# Grid search for hyperparameter tuning
search_grid <- expand.grid(
  max_depth = c(3, 6),
  eta = c(0.01, 0.1),
  colsample_bytree = c(0.5, 0.7)
)

best_auc <- 0
best_params <- list()

for (i in 1:nrow(search_grid)) {
  params <- list(
    objective = "multi:softprob",
    eval_metric = "mlogloss",
    num_class = 3,
    max_depth = search_grid$max_depth[i],
    eta = search_grid$eta[i],
    colsample_bytree = search_grid$colsample_bytree[i]
  )
  
  cv_results <- xgb.cv(
    params = params,
    data = dtrain,
    nfold = 5,
    nrounds = 100,
    early_stopping_rounds = 10,
    verbose = TRUE
  )
  
  mean_auc <- max(cv_results$evaluation_log$test_mlogloss_mean)
  
  if (mean_auc > best_auc) {
    best_auc <- mean_auc
    best_params <- params
    best_nrounds <- cv_results$best_iteration
  }
}

# Train the final model with the best parameters
final_model <- xgb.train(
  params = best_params,
  data = dtrain,
  nrounds = best_nrounds
)

Output:

[1]    train-mlogloss:0.983851+0.006663    test-mlogloss:0.987265+0.009371 
Multiple eval metrics are present. Will use test_mlogloss for early stopping.
Will train until test_mlogloss hasn't improved in 10 rounds.

[2]    train-mlogloss:0.878013+0.007136    test-mlogloss:0.884146+0.007905 
[3]    train-mlogloss:0.796634+0.015013    test-mlogloss:0.808550+0.021133 
[4]    train-mlogloss:0.717974+0.013423    test-mlogloss:0.736113+0.013818 
[5]    train-mlogloss:0.653546+0.016296    test-mlogloss:0.675885+0.017799 
[6]    train-mlogloss:0.596697+0.018427    test-mlogloss:0.622178+0.023205 
[7]    train-mlogloss:0.545010+0.015754    test-mlogloss:0.571462+0.020152 
[8]    train-mlogloss:0.497209+0.013964    test-mlogloss:0.525631+0.020119 
[9]    train-mlogloss:0.456389+0.015891    test-mlogloss:0.488164+0.016598 
[10]    train-mlogloss:0.417232+0.014707    test-mlogloss:0.452470+0.020607 
[11]    train-mlogloss:0.383913+0.017773    test-mlogloss:0.423728+0.021391 
[12]    train-mlogloss:0.352606+0.015181    test-mlogloss:0.396152+0.021836 
[13]    train-mlogloss:0.324013+0.013673    test-mlogloss:0.367447+0.022506 
[14]    train-mlogloss:0.299813+0.012686    test-mlogloss:0.343159+0.024509 
[15]    train-mlogloss:0.276713+0.011857    test-mlogloss:0.319813+0.025241 
[16]    train-mlogloss:0.256142+0.011000    test-mlogloss:0.301409+0.027859 
[17]    train-mlogloss:0.236637+0.010686    test-mlogloss:0.283405+0.029481 
[18]    train-mlogloss:0.219567+0.009666    test-mlogloss:0.268679+0.031260 
[19]    train-mlogloss:0.204322+0.009568    test-mlogloss:0.254008+0.032928 
[20]    train-mlogloss:0.190355+0.009219    test-mlogloss:0.243043+0.035494 
[21]    train-mlogloss:0.177822+0.008835    test-mlogloss:0.231704+0.037772 
[22]    train-mlogloss:0.166672+0.007838    test-mlogloss:0.222843+0.040372 
[23]    train-mlogloss:0.155972+0.007457    test-mlogloss:0.213581+0.043389 
[24]    train-mlogloss:0.146813+0.007017    test-mlogloss:0.206327+0.044050 
[25]    train-mlogloss:0.138563+0.006575    test-mlogloss:0.200042+0.044902 
[26]    train-mlogloss:0.130210+0.006502    test-mlogloss:0.193591+0.047043 
[27]    train-mlogloss:0.123683+0.006875    test-mlogloss:0.188267+0.047222 
[28]    train-mlogloss:0.116948+0.007335    test-mlogloss:0.184278+0.048900 
[29]    train-mlogloss:0.111106+0.007952    test-mlogloss:0.179855+0.050081 
[30]    train-mlogloss:0.105462+0.008591    test-mlogloss:0.175083+0.050175 
[31]    train-mlogloss:0.100460+0.008470    test-mlogloss:0.171258+0.049444 
[32]    train-mlogloss:0.095373+0.008036    test-mlogloss:0.166830+0.050722 
[33]    train-mlogloss:0.090179+0.007897    test-mlogloss:0.164154+0.052903 
[34]    train-mlogloss:0.086006+0.007739    test-mlogloss:0.162114+0.055078 
[35]    train-mlogloss:0.081960+0.007942    test-mlogloss:0.160610+0.057457 
[36]    train-mlogloss:0.078338+0.007718    test-mlogloss:0.159158+0.058757 
[37]    train-mlogloss:0.075079+0.007464    test-mlogloss:0.156747+0.059068 
[38]    train-mlogloss:0.072091+0.007289    test-mlogloss:0.155119+0.060228 
[39]    train-mlogloss:0.068796+0.007076    test-mlogloss:0.154012+0.061161 
[40]    train-mlogloss:0.066199+0.007249    test-mlogloss:0.153072+0.062036 
[41]    train-mlogloss:0.064103+0.007136    test-mlogloss:0.151745+0.062377 
[42]    train-mlogloss:0.061883+0.006879    test-mlogloss:0.150824+0.063542 
[43]    train-mlogloss:0.059500+0.006803    test-mlogloss:0.150588+0.065450 
[44]    train-mlogloss:0.057294+0.006589    test-mlogloss:0.149414+0.065131 
[45]    train-mlogloss:0.055483+0.006559    test-mlogloss:0.148611+0.065323 
[46]    train-mlogloss:0.053759+0.006323    test-mlogloss:0.147631+0.065561 
[47]    train-mlogloss:0.052255+0.006161    test-mlogloss:0.147121+0.066141 
[48]    train-mlogloss:0.050774+0.006213    test-mlogloss:0.146550+0.066843 
[49]    train-mlogloss:0.049445+0.006151    test-mlogloss:0.146102+0.066932 
[50]    train-mlogloss:0.048066+0.006018    test-mlogloss:0.145531+0.066973 
[51]    train-mlogloss:0.046986+0.005957    test-mlogloss:0.144657+0.066806 
[52]    train-mlogloss:0.045785+0.005850    test-mlogloss:0.145033+0.067564 
[53]    train-mlogloss:0.044669+0.005634    test-mlogloss:0.143860+0.067970 
[54]    train-mlogloss:0.043621+0.005608    test-mlogloss:0.143428+0.068309 
[55]    train-mlogloss:0.042637+0.005325    test-mlogloss:0.142644+0.068432 
[56]    train-mlogloss:0.041743+0.005417    test-mlogloss:0.142826+0.068745 
[57]    train-mlogloss:0.040911+0.005341    test-mlogloss:0.143199+0.068868 
[58]    train-mlogloss:0.039920+0.005289    test-mlogloss:0.143989+0.069813 
[59]    train-mlogloss:0.039135+0.005177    test-mlogloss:0.144111+0.070286 
[60]    train-mlogloss:0.038332+0.005010    test-mlogloss:0.143885+0.070162 
[61]    train-mlogloss:0.037630+0.004899    test-mlogloss:0.144392+0.070564 
[62]    train-mlogloss:0.036975+0.004718    test-mlogloss:0.144930+0.071065 
[63]    train-mlogloss:0.036509+0.004567    test-mlogloss:0.145638+0.071851 
[64]    train-mlogloss:0.036061+0.004484    test-mlogloss:0.145488+0.072193 
[65]    train-mlogloss:0.035611+0.004447    test-mlogloss:0.146268+0.072840 
Stopping. Best iteration:
[55]    train-mlogloss:0.042637+0.005325    test-mlogloss:0.142644+0.068432
  1. Define Initial Parameters: Specify the fixed parameters and set up the xgb.cv function.
  2. Perform Cross-Validation: Use xgb.cv to determine the best number of rounds.
  3. Hyperparameter Tuning: Implement a grid search loop to find the optimal hyperparameters.
  4. Train Final Model: Use xgb.train with the optimal parameters and number of rounds.

By following these steps, you can efficiently perform cross-validation, tune hyperparameters, and train an optimized XGBoost model in R.

Conclusion

xgb.cv in R is a powerful tool for tuning hyperparameters and optimizing XGBoost models using cross-validation. By systematically evaluating different combinations of parameters and identifying the optimal set, xgb.cv helps improve model performance and generalization.

Once the optimal parameters are determined, they are passed into xgb.train for model training. This seamless transfer ensures that the final model is trained with the best hyperparameters, resulting in improved predictive accuracy and robustness. Understanding the workflow of xgb.cv and how it passes optimal parameters to xgb.train is essential for effectively tuning XGBoost models and building high-performance machine learning solutions in R.



Tags:
#AI-ML-DS With R #Data Science Blogathon 2024 #AI-ML-DS #Blogathon #Machine Learning #Machine Learning

Set or View the Graphics Palette in R Programming - palette() Function

Python Numpy

Contact Us

Categories
Learn the most popular programming languages in the world.