Terminology or Notations

Following notations must be kept in mind while understanding the Weight Initialization Techniques. These notations may vary at different publications. However, the ones used here are the most common, usually found in research papers.

fan_in = Number of input paths towards the neuron

fan_out = Number of output paths towards the neuron

Example: Consider the following neuron as a part of a Deep Neural Network.

For the above neuron,

fan_in = 3 (Number of input paths towards the neuron)

fan_out = 2 (Number of output paths towards the neuron)

While building and training neural networks, it is crucial to initialize the weights appropriately to ensure a model with high accuracy. If the weights are not correctly initialized, it may give rise to the Vanishing Gradient problem or the Exploding Gradient problem. Hence, selecting an appropriate weight initialization strategy is critical when training DL models. In this article, we will learn some of the most common weight initialization techniques, along with their implementation in Python using Keras in TensorFlow.

As pre-requisites, the readers of this article are expected to have a basic knowledge of weights, biases and activation functions. In order to understand what this all, you are and what role they play in Deep Neural Networks – you are advised to read through the article Deep Neural Network With L – Layers