Transform Text Features to Numerical Features with CatBoost
Handling text and category data is essential to machine learning to create correct prediction models. Yandex’s gradient boosting library, CatBoost, performs very well. It provides sophisticated methods to convert text characteristics into numerical ones and supports categorical features natively, both of which may greatly enhance model performance. This article will focus on how to transform text features into numerical features using CatBoost, enhancing the model’s predictive power.
Table of Content
- Text Processing in CatBoost
- Steps to Transform Text Features to Numerical Features
- 1. Loading and Storing Text Features
- 2. Preprocessing Text Features
- 3. Calculating New Features
- 4. Training the Model
- Text Features to Numerical Features using CatBoost : Implementation
Text Processing in CatBoost
Text features in CatBoost are used to build new numeric features. These features are essential for tasks involving natural language processing (NLP), where raw text data needs to be converted into a format that machine learning models can understand and process effectively.
There are many processes involved in CatBoost’s text processing:
- Tokenization: The process of dividing text into relevant tokens.
- Embedding: Changing tokens into vectors of numbers.
- Aggregation: Creating fixed-length numerical characteristics by summing these vectors.
Handling Text Features in CatBoost
When dealing with text features, it is crucial to ensure that the order of columns in the training and test datasets matches. This can be managed by using the Pool
method in CatBoost, where columns can be added by name.
Example of Using Text Features:
model.fit(x_train, y_train, text_features=['text'])
For prediction, ensure the text features are correctly specified:
preds_class = model.predict(X_test)
Steps to Transform Text Features to Numerical Features
1. Loading and Storing Text Features
Text features are loaded into CatBoost similarly to other feature types. They can be specified in the column descriptions file or directly in the Python package using the text_features
parameter.
2. Preprocessing Text Features
CatBoost uses dictionaries and tokenizers to preprocess text features. The dictionaries define how text data is converted into tokens, while tokenizers break down the text into these tokens.
Example of a Dictionary:
dictionaries = [{
'dictionaryId': 'Unigram',
'max_dictionary_size': '50000',
'gram_count': '1',
}, {
'dictionaryId': 'Bigram',
'max_dictionary_size': '50000',
'gram_count': '2',
}]
Example of a Tokenizer:
tokenizers = [{
'tokenizerId': 'Space',
'delimiter': ' ',
}]
3. Calculating New Features
Feature calculators (feature calcers) are used to generate new numeric features from the preprocessed text data. These calculators can include methods like Bag of Words (BoW), Naive Bayes, and others.
Example of Feature Calcers:
feature_calcers = [
'BoW:top_tokens_count=1000',
'NaiveBayes',
]
4. Training the Model
Once the text features are preprocessed and new numeric features are calculated, they are passed to the regular CatBoost training algorithm.
Text Features to Numerical Features using CatBoost : Implementation
Step 1: Install CatBoost and Import CatBoost
Ensure you have CatBoost installed:
!pip install catboost
Importing CatBoost
from catboost import CatBoostClassifier, Pool
import pandas as pd
Step 2: Prepare Dataset
We’ll illustrate the procedure using an example dataset. Here, categorical characteristics like “City” and “Weather” are present in the dataset:
data = {
'City': ['New York', 'London', 'Tokyo', 'New York', 'Tokyo'],
'Weather': ['Sunny', 'Rainy', 'Sunny', 'Snowy', 'Rainy'],
'Label': [1, 0, 1, 0, 0]
}
df = pd.DataFrame(data)
Step 3: Define Features and Target
Determine the target variable and its characteristics:
X = df[['City', 'Weather']]
y = df['Label']
Step 4: Initialize and Train the Model
Establish categorical characteristics and set the CatBoostClassifier’s initialization, To manage the data and indicate which characteristics are categorical, create a Pool object as follows:
categorical_features = ['City', 'Weather']
model = CatBoostClassifier(iterations=100, depth=3, learning_rate=0.1, loss_function='Logloss')
train_pool = Pool(data=X, label=y, cat_features=categorical_features)
model.fit(train_pool)
Step 5: View Transformed Features
During training, CatBoost internally modifies the category characteristics. You may access the feature importances in order to examine the altered features:
importances = model.get_feature_importance(train_pool, prettified=True)
print(importances)
Output:
Feature Id Importances
0 City 82.857487
1 Weather 17.142513
Conclusion
Transforming text features into numerical features in CatBoost involves preprocessing text data using dictionaries and tokenizers, calculating new numeric features with feature calcers, and then training the model. This process enhances the model’s ability to handle text data effectively, making CatBoost a robust tool for NLP tasks. By following the steps outlined in this article, you can leverage CatBoost’s capabilities to transform and utilize text features in your machine learning models, improving their predictive performance.
Contact Us