Visualization of Adam

Mathematical Intuition Behind Adam’s Gradient Descent

To visualize the optimization process using the Adam algorithm, we can plot the trajectory of the candidate points as they move toward the minimum of the objective function. Here’s an example of how you can visualize the optimization process using Matplotlib:

Python3

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
 
# Objective function
def objective(x, y):
    return x ** 2.0 + y ** 2.0
 
# Derivative of the objective function
def derivative(x, y):
    return np.array([2.0 * x, 2.0 * y])
 
# Gradient descent algorithm with Adam
def adam(objective, derivative, bounds, n_iter,
         alpha, beta1, beta2, eps=1e-8):
    # Generate an initial point
    x = bounds[:, 0] + np.random.rand(len(bounds))\
    * (bounds[:, 1] - bounds[:, 0])
    scores = []
    trajectory = []
 
    # Initialize first and second moments
    m = np.zeros(bounds.shape[0])
    v = np.zeros(bounds.shape[0])
 
    # Run the gradient descent updates
    for t in range(n_iter):
        # Calculate gradient g(t)
        g = derivative(x[0], x[1])
 
        # Build a solution one variable at a time
        for i in range(x.shape[0]):
            # m(t) = beta1 * m(t-1) + (1 - beta1) * g(t)
            m[i] = beta1 * m[i] + (1.0 - beta1) * g[i]
            # v(t) = beta2 * v(t-1) + (1 - beta2) * g(t)^2
            v[i] = beta2 * v[i] + (1.0 - beta2) * g[i] ** 2
            # mhat(t) = m(t) / (1 - beta1(t))
            mhat = m[i] / (1.0 - beta1 ** (t + 1))
            # vhat(t) = v(t) / (1 - beta2(t))
            vhat = v[i] / (1.0 - beta2 ** (t + 1))
            # x(t) = x(t-1) - alpha * mhat(t) / (sqrt(vhat(t)) + eps)
            x[i] = x[i] - alpha * mhat / (np.sqrt(vhat) + eps)
 
        # Evaluate candidate point
        score = objective(x[0], x[1])
        scores.append(score)
        trajectory.append(x.copy())
 
    return x, scores, trajectory
 
# Define the range for input
bounds = np.array([[-1.0, 1.0], [-1.0, 1.0]])
 
# Define the total number of iterations
n_iter = 60
 
# Set the step size
alpha = 0.02
 
# Set the factor for average gradient
beta1 = 0.8
 
# Set the factor for average squared gradient
beta2 = 0.999
 
# Perform the gradient descent search with Adam
best, scores, trajectory = adam(objective, derivative,
                                bounds, n_iter, alpha,
                                beta1, beta2)
 
# Generate a grid of points for visualization
x = np.linspace(bounds[0, 0], bounds[0, 1], 100)
y = np.linspace(bounds[1, 0], bounds[1, 1], 100)
X, Y = np.meshgrid(x, y)
Z = objective(X, Y)
 
# Plot the optimization trajectory
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(X, Y, Z, cmap='viridis', alpha=0.5)
ax.scatter(best[0], best[1], objective(best[0], best[1]),
           color='red', label='Best')
ax.plot([point[0] for point in trajectory],
        [point[1] for point in trajectory], scores,
        color='blue', label='Trajectory')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Objective')
ax.legend()
 
# Show the plot
plt.show()

Output:

Global Minima Using Adam Optimization Algorithm

The above code generates a 3D plot of the objective function and overlays the optimization trajectory of the candidate points. The best solution found is shown as a red dot, and the trajectory is shown as a blue line connecting the candidate points.

Advantages of the Adam Algorithm

Effective Handling of Sparse Gradients: Adam performs well on datasets with sparse gradients, making it suitable for tasks involving noisy or irregularly sampled data.
Default Hyperparameter Values: Adam’s default hyperparameter values often yield good results across a wide range of problems, reducing the need for extensive hyperparameter tuning.
Computational Efficiency: Adam is computationally efficient, making it suitable for large-scale optimization tasks with a high-dimensional parameter space.
Memory Efficiency: Adam requires relatively low memory compared to some other optimization algorithms, making it memory-efficient, particularly when dealing with large datasets.
Suitable for Large Datasets: Adam is well-suited for optimization problems involving large datasets, as it efficiently updates the parameters based on the accumulated gradients.

Disadvantages of the Adam Algorithm

Convergence Issues in Certain Cases: Adam may not converge to an optimal solution in some cases, particularly in regions with complex or irregular geometry. This limitation has motivated the development of alternative variants such as AMSGrad.
Weight Decay Problem: Adam can suffer from a weight decay problem, where the weight decay term overly penalizes the parameters during optimization. This issue has been addressed in variations like AdamW, which incorporate additional techniques to mitigate the problem.
Continuous Algorithm Improvements: Recent advancements in optimization algorithms have demonstrated better performance and faster convergence rates compared to Adam, indicating that there are alternatives that may outperform it in certain scenarios.

How to Implement Adam Gradient Descent from Scratch using Python?

Grade descent is an extensively used optimization algorithm in machine literacy and deep literacy. It’s used to minimize the cost or loss function of a model by iteratively confirming the model’s parameters grounded on the slants of the cost function with respect to those parameters. One variant of gradient descent that has gained popularity is the Adam optimization algorithm. Adam combines the benefits of AdaGrad and RMSProp to achieve effective and adaptive learning rates.

Similar Reads

Gradient Descent: An iterative optimization algorithm used to find the minimum of a function by iteratively adjusting the parameters in the direction of the steepest descent of the gradient. Learning Rate: A hyperparameter that determines the step size at each iteration of gradient descent. It controls how much the parameters are updated based on the computed gradients. Objective Function: The function that we aim to minimize or maximize. In the context of optimization, it is usually a cost function or a loss function. Derivative: The derivative of a function represents its rate of change at a particular point. In the context of gradient descent, the derivative (or gradient) provides the direction and magnitude of the steepest ascent or descent. Local Minimum: A point in the optimization landscape where the objective function has the lowest value in the vicinity. It may not be the global minimum, which is the absolute lowest point in the entire optimization landscape. Global Minimum: The lowest point in the entire optimization landscape, indicating the optimal solution to the problem. Momentum: In the context of optimization algorithms, momentum refers to the inclusion of past gradients to determine the current update. It helps accelerate convergence and overcome local optima. Adam (Adaptive Moment Estimation): An optimization algorithm that extends gradient descent by utilizing adaptive learning rates and momentum. It maintains a running average of past gradients and squared gradients to compute adaptive learning rates for each parameter. Beta1 and Beta2: Hyperparameters in the Adam algorithm that control the exponential decay rates for the estimation of the first and second moments of the gradients, respectively. Beta1 controls the momentum-like effect, while Beta2 controls the influence of the squared gradients. Epsilon (eps): A small constant added to the denominator in the Adam algorithm to prevent division by zero and ensure numerical stability....

Tags:

#AI-ML-DS #Machine Learning #Machine Learning