#batchnorm

via WIRED After getting rejecting initially, the US patent office, recently, has granted the ownership of batch normalisation to Google with an expiration date marked as 2038.  Here is a timeline of the journey of the BatchNorm patent application: 2015-01-28 Priority to US201562108984P 2016-01-28 Application filed by Google LLC 2016-07-28 Publication of US20160217368A1 2019-09-17 Publication of US10417562B2 2019-09-17 Application granted 2019-10-15 Application status is Active 2038-01-01 Adjusted expiration Google does patent many of its products but there is a reason why batch normalisation patent gets all the attention.  Why BatchNorm Touched A Nerve In the original BatchNorm paper, the authors, Sergey Ioffe and Christian Szegedy of Google introduced this method to address a phenomenon called internal covariate shift.  This occurs because the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change.  This slows down the training by requiring lower learning rates and careful parameter initialisation. This makes training the models harder. The introduction of batch normalised networks helped achieve state-of-the-art accuracies with 14 times fewer training steps.  This reduction in training duty led to the emergence of many improvements within the machine learning community.  In spite of voicing for the democratisation of technology, companies like Google are rushing to claim ownership of advanced approaches in domains like AI. However, before jumping the bandwagon of critics, one should know that machine learning techniques like batch normalisation are a product of Google.  The researchers working at Google used the resources available at Google to develop new frameworks to optimise machine learning applications.  So, claiming ownership does sound right as it has been classified as a defensive patenting. One argument that is being made for Google is that it is better if the creators have the ownership rather than a patent troll claiming ownership and creating hurdles amidst lawsuits.  However, a large half of the machine learning community is still sceptical of this consistent claim of ownership. Few members of the most happening machine learning forums like that of Reddit likened this whole event to a loaded gun that can backfire. Working Around This Trilemma There is no denying the fact that Google AI has been pioneering even before machine learning became a worldwide phenomenon. Their technology is being used to power up lives and promote growth.  The use of BatchNorm has literally become a norm for many of the previous applications. This also is the reason for the rising scepticism amongst the practitioners. For instance, if a computer vision startup has used this technique to train one of its neural networks then should it be checking the mailbox for a probable lawsuit forever?  That said, ever since the news broke out that Google has eyed batch normalisation technique, the alternative approaches to batch norm technique have gained traction. Here few methods that look promising: Fixup Initialisation: Fixed-update initialisation (Fixup) was aimed at solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialisation. Using Weight Normalisation: Weight normalisation accelerates the convergence of stochastic gradient descent optimisation by re-parameterising weight vectors in neural networks.  General Hamming Network (GHN): The researchers at Nokia technologies in their work illustrated that the celebrated batch normalisation (BN) technique actually adapts the “normalised” bias such that it approximates the rightful bias induced by the generalised hamming distance. Group Normalisation (GN): GN divides the channels into groups and computes within each group the mean and variance for normalisation. GN’s computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. Switchable Normalisation (SN): Switchable Normalisation (SN) learns to select different normalisers for different normalisation layers of a deep neural network. Attentive Normalisation (AN): Attentive Normalisation(AN) is a novel and lightweight integration of feature normalisation and feature channel-wise attention. These are only a few of the many approaches that have surfaced in recent times and we can safely assume that there will be more coming up from this space.  Read the full article

When using keras as a simplified interface to tensorflow, some Keras layers (stateful RNNs and BatchNormalization layers) have internal updates that need to be run as part of each training step. There are stored as a list of tensor tuples, layer.updates. We should generate assign ops for those, to be run at each training step. Here's an example:

If you are comfortable with TensorFlow’s underlying graph/ops mechanism, the note is fairly straight-forward. If not, here’s a simple way to think of it: when you execute an operation (such as train_step), only the subgraph components relevant to train_step will be executed. Unfortunately, the update_moving_averages operation is not a parent of train_step in the computational graph, so we will never update the moving averages! To get around this, we have to explicitly tell the graph: Hey graph, update the moving averages before you finish the training step!