Dynamic computation graph in Pytorch
Properties of nodes & edges: The nodes represent the data(in form of tensors) and the edges represent the operations applied to the input data.
For the equations given in the Introduction, we can keep the following things in mind while implementing it in Pytorch:
- Since everything in Pytorch is created dynamically, we don’t need any placeholders and can define our inputs and operations on the fly.
- After defining the inputs and computing the output ‘c’, we call the backward() method, which calculates the corresponding partial derivatives with respect to the two inputs accessible through the .grad specifier.
Now let’s check out a code example to verify our findings:
Python3
# Importing torch import torch # Initializing input tensors a = torch.tensor(15.0, requires_grad=True) b = torch.tensor(20.0, requires_grad=True) # Computing the output c = a * b # Computing the gradients c.backward() # Collecting the output gradient of the # output with respect to the input 'a' derivative_out_a = a.grad # Collecting the output gradient of the # output with respect to the input 'b' derivative_out_b = b.grad # Displaying the outputs print(f'c = {c}') print(f'Derivative of c with respect to a = {derivative_out_a}') print(f'Derivative of c with respect to b = {derivative_out_b}') |
Output:
c = 300.0 Derivative of c with respect to a = 20.0 Derivative of c with respect to b = 15.0
As we can see, the output matches correctly with our calculations in the Introduction section, thus indicating successful completion. The dynamic structure is evident from the code. We can see that all the inputs and outputs can be accessed and changed during the runtime only, which is entirely different from the approach used by Tensorflow.
Advantages:
- Scalability to different dimensional inputs: Scales very well for different dimensional inputs as a new pre-processing layer can be dynamically added to the network itself.
- Ease in debugging: These are very easy to debug and are one of the reasons why many people are shifting from Tensorflow to Pytorch. As the nodes are created dynamically before any information flows through them, the error becomes very easy to spot as the user is in complete control of the variables used in the training process.
Disadvantages:
- Allows very little room for graph optimization because a new graph needs to be created for each training instance/batch.
Dynamic vs Static Computational Graphs – PyTorch and TensorFlow
TensorFlow and Pytorch are two of the most popular deep learning libraries recently. Both libraries have developed their respective niches in mainstream deep learning with excellent documentation, tutorials, and, most importantly, an exuberant and supportive community behind them.
Contact Us