Latest

Derivation of Glorot initialisation

2024-03-08

Contents

• 1 Introduction
• 2 Intended audience
• 3 What is Glorot initialisation?
• 4 Brief outline of Glorot et al's derivation
• 5 My commentary on Glorot et al's derivation
• 5.1 Prelude: Desirable statistical properties of the activation vectors and cost function gradients of a neural net at initialisation
• 5.2 Assumptions used in the derivation
• 5.3 Note: Formatting for Glorot et al's work vs. my commentary
• 5.4 Start of the derivation
• 6 Conclusion
• 7 References

1 Introduction

Glorot initialisation (AKA Xavier initialisation) is today the most popular method for initialising the weights of deep neural networks. It was introduced by Xavier Glorot and Yoshua Bengio in a landmark paper [1] published in 2010. Since then, Glorot initialisation has become the default initialisation method in popular deep learning libraries such as Keras/TensorFlow.

In this post, I will derive Glorot initialisation from first principles via a commentary on Glorot et al's paper, referring to his original derivation verbatim and filling in any details where more explanation could be provided for clarity. By keeping consistent with his notation, my intention is that this post can be read directly alongside his paper.