Cross Entropy for Multi-class Classification
Now that we have a basic understanding of the theory and the mathematical formulation of the cross entropy function now let’s try to work on a sample problem to get a feel of how the value for the cross entropy cost function is calculated.
Example 1: Actual Probabilities: [0, 1, 0, 0] Predicted Probabilities: [0.21, 0.68, 0.09, 0.10] Cross Entropy = - 1 * log(0.68) = 0.167
From the above example, we can observe that all the three value for which the actual probabilities was equal to zero becomes zero and the value of the cross entropy depends upon the predicted value for the class whose probability is one.
Example 2: Actual Probabilities: [0, 1, 0, 0] Predicted Probabilities: [0.12, 0.28, 0.34, 0.26] Cross Entropy = - 1 * log(0.28) = 0.558 Example 3: Actual Probabilities: [0, 1, 0, 0] Predicted Probabilities: [0.05, 0.80, 0.09, 0.06] Cross Entropy = - 1 * log(0.80) = 0.096
One may say that all three examples are more or less the same but no there is a subtle difference between all three which is comparable in that as the predicted probability is close to the actual probability the value of the cross entropy decreases but as the predicted probability deviates from the actual probability value of the cross entropy function shoots up.
Cross-Entropy Cost Functions used in Classification
Cost functions play a crucial role in improving a Machine Learning model’s performance by being an integrated part of the gradient descent algorithm which helps us optimize the weights of a particular model. In this article, we will learn about one such cost function which is the cross-entropy function which is generally used for classification problems.
Contact Us