You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by du...@apache.org on 2017/03/30 21:38:37 UTC
incubator-systemml git commit: [SYSTEMML-1409][SYSTEMML-1410] New
Batch Normalization Layers
Repository: incubator-systemml
Updated Branches:
refs/heads/master 0f4571810 -> c944ad1dc
[SYSTEMML-1409][SYSTEMML-1410] New Batch Normalization Layers
This commit adds a new batch normalization layer, `batch_norm`, and a
new spatial batch normalization layer, `spatial_batch_norm`.
The batch normalization layer uses the per-feature sample mean and
per-feature uncorrected sample variance during training to
normalize each feature of the input data.
The spatial batch normalization layer uses the per-channel sample mean
and per-channel uncorrected sample variance during training to
normalize each channel of the input data.
Additionally, these layers introduce learnable parameters (gamma, beta)
to control the amount of normalization.
y = ((x-mean) / sqrt(var+eps)) * gamma + beta
Finally, these implementations maintain exponential moving averages of
the mean and variance during training for use during testing.
Closes #444.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/c944ad1d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/c944ad1d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/c944ad1d
Branch: refs/heads/master
Commit: c944ad1dca3dd0110c874e227f7511c9a182f26a
Parents: 0f45718
Author: Mike Dusenberry <mw...@us.ibm.com>
Authored: Thu Mar 30 14:37:11 2017 -0700
Committer: Mike Dusenberry <mw...@us.ibm.com>
Committed: Thu Mar 30 14:37:11 2017 -0700
----------------------------------------------------------------------
.../SystemML-NN/nn/layers/batch_norm.dml | 208 ++++++++++++++++
.../nn/layers/spatial_batch_norm.dml | 235 +++++++++++++++++++
.../staging/SystemML-NN/nn/test/grad_check.dml | 216 +++++++++++++++++
scripts/staging/SystemML-NN/nn/test/test.dml | 132 ++++++++++-
scripts/staging/SystemML-NN/nn/test/tests.dml | 9 +
scripts/staging/SystemML-NN/nn/util.dml | 19 ++
6 files changed, 816 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/c944ad1d/scripts/staging/SystemML-NN/nn/layers/batch_norm.dml
----------------------------------------------------------------------
diff --git a/scripts/staging/SystemML-NN/nn/layers/batch_norm.dml b/scripts/staging/SystemML-NN/nn/layers/batch_norm.dml
new file mode 100644
index 0000000..d332e8c
--- /dev/null
+++ b/scripts/staging/SystemML-NN/nn/layers/batch_norm.dml
@@ -0,0 +1,208 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+/*
+ * Batch normalization layer.
+ */
+forward = function(matrix[double] X, matrix[double] gamma, matrix[double] beta,
+ string mode, matrix[double] ema_mean, matrix[double] ema_var,
+ double mu, double epsilon)
+ return (matrix[double] out, matrix[double] ema_mean_upd, matrix[double] ema_var_upd,
+ matrix[double] cache_mean, matrix[double] cache_var, matrix[double] cache_norm) {
+ /*
+ * Computes the forward pass for a batch normalization layer.
+ *
+ * A batch normalization layer uses the per-feature sample mean and
+ * per-feature uncorrected sample variance during training to
+ * normalize each feature of the input data. Additionally, it
+ * introduces learnable parameters (gamma, beta) to control the
+ * amount of normalization.
+ *
+ * y = ((x-mean) / sqrt(var+eps)) * gamma + beta
+ *
+ * This implementation maintains exponential moving averages of the
+ * mean and variance during training for use during testing.
+ *
+ * Reference:
+ * - Batch Normalization: Accelerating Deep Network Training by
+ * Reducing Internal Covariate Shift, S. Ioffe & C. Szegedy, 2015
+ * - https://arxiv.org/abs/1502.03167
+ *
+ * Inputs:
+ * - X: Input data matrix, of shape (N, D).
+ * - gamma: Scale parameters, of shape (1, D).
+ * - beta: Shift parameters, of shape (1, D).
+ * - mode: 'train' or 'test' to indicate if the model is currently
+ * being trained or tested. During training, the current batch
+ * mean and variance will be used to normalize the inputs, while
+ * during testing, the exponential average of the mean and
+ * variance over all previous batches will be used.
+ * - ema_mean: Exponential moving average of the mean, of
+ * shape (1, D).
+ * - ema_var: Exponential moving average of the variance, of
+ * shape (1, D).
+ * - mu: Momentum value for moving averages.
+ * Typical values are in the range of [0.9, 0.999].
+ * - epsilon: Smoothing term to avoid divide by zero errors.
+ * Typical values are in the range of [1e-5, 1e-3].
+ *
+ * Outputs:
+ * - out: Outputs, of shape (N, D).
+ * - ema_mean_upd: Updated exponential moving average of the mean,
+ * of shape (1, D).
+ * - ema_var_upd: Updated exponential moving average of the variance,
+ * of shape (1, D).
+ * - cache_mean: Cache of the batch mean, of shape (1, D).
+ * Note: This is used for performance during training.
+ * - cache_var: Cache of the batch variance, of shape (1, D).
+ * Note: This is used for performance during training.
+ * - cache_norm: Cache of the normalized inputs, of shape (N, D).
+ * Note: This is used for performance during training.
+ */
+ N = nrow(X)
+
+ if(mode == 'train') {
+ # Compute feature-wise mean and variance
+ mean = colMeans(X) # shape (1, D)
+ # var = (1/N) * colSums((X-mean)^2)
+ var = colVars(X) * ((N-1)/N) # compute uncorrected variance, of shape (1, D)
+ # Update moving averages
+ ema_mean_upd = mu*ema_mean + (1-mu)*mean
+ ema_var_upd = mu*ema_var + (1-mu)*var
+ }
+ else {
+ # Use moving averages of mean and variance during testing
+ mean = ema_mean
+ var = ema_var
+ ema_mean_upd = ema_mean
+ ema_var_upd = ema_var
+ }
+
+ # Normalize, shift, and scale
+ # norm = (X-mean)*(var+epsilon)^(-1/2)
+ norm = (X-mean) / sqrt(var+epsilon) # shape (N, D)
+ out = norm*gamma + beta # shape (N, D)
+
+ # Save variable for backward pass
+ cache_mean = mean
+ cache_var = var
+ cache_norm = norm
+}
+
+backward = function(matrix[double] dout, matrix[double] out,
+ matrix[double] ema_mean_upd, matrix[double] ema_var_upd,
+ matrix[double] cache_mean, matrix[double] cache_var, matrix[double] cache_norm,
+ matrix[double] X, matrix[double] gamma, matrix[double] beta,
+ string mode, matrix[double] ema_mean, matrix[double] ema_var,
+ double mu, double epsilon)
+ return (matrix[double] dX, matrix[double] dgamma, matrix[double] dbeta) {
+ /*
+ * Computes the backward pass for a batch normalization layer.
+ *
+ * Inputs:
+ * - dout: Derivatives from upstream, of shape (N, D).
+ * - out: Outputs from the forward pass, of shape (N, D).
+ * - ema_mean_upd: Updated exponential moving average of the mean
+ * from the forward pass, of shape (1, D).
+ * - ema_var_upd: Updated exponential moving average of the variance
+ * from the forward pass, of shape (1, D).
+ * - cache_mean: Cache of the batch mean from the forward pass, of
+ * shape (1, D). Note: This is used for performance during
+ * training.
+ * - cache_var: Cache of the batch variance from the forward pass,
+ * of shape (1, D). Note: This is used for performance during
+ * training.
+ * - cache_norm: Cache of the normalized inputs from the forward
+ * pass, of shape (N, D). Note: This is used for performance
+ * during training.
+ * - X: Input data matrix to the forward pass, of shape (N, D).
+ * - gamma: Scale parameters, of shape (1, D).
+ * - beta: Shift parameters, of shape (1, D).
+ * - mode: 'train' or 'test' to indicate if the model is currently
+ * being trained or tested. During training, the current batch
+ * mean and variance will be used to normalize the inputs, while
+ * during testing, the exponential average of the mean and
+ * variance over all previous batches will be used.
+ * - ema_mean: Exponential moving average of the mean, of
+ * shape (1, D).
+ * - ema_var: Exponential moving average of the variance, of
+ * shape (1, D).
+ * - mu: Momentum value for moving averages.
+ * Typical values are in the range of [0.9, 0.999].
+ * - epsilon: Smoothing term to avoid divide by zero errors.
+ * Typical values are in the range of [1e-5, 1e-3].
+ *
+ * Outputs:
+ * - dX: Gradient wrt X, of shape (N, D).
+ * - dgamma: Gradient wrt W, of shape (1, D).
+ * - dbeta: Gradient wrt b, of shape (1, D).
+ *
+ */
+ N = nrow(X)
+ mean = cache_mean
+ var = cache_var
+ norm = cache_norm
+ centered = X-mean
+
+ if (mode == 'train') {
+ # Compute gradients during training
+ dgamma = colSums(norm*dout) # shape (1, D)
+ dbeta = colSums(dout) # shape (1, D)
+ dnorm = dout * gamma # shape (N, D)
+ dvar = (-1/2) * colSums(centered * (var+epsilon)^(-3/2) * dnorm) # shape (1, D)
+ dmean = colSums((-dnorm/sqrt(var+epsilon)) + ((-2/N)*centered*dvar)) # shape (1, D)
+ dX = (dnorm/sqrt(var+epsilon)) + ((2/N)*centered*dvar) + ((1/N)*dmean) # shape (N, D)
+ }
+ else {
+ # Compute gradients during testing
+ dgamma = colSums(norm*dout) # shape (1, D)
+ dbeta = colSums(dout) # shape (1, D)
+ dnorm = dout * gamma # shape (N, D)
+ dX = dnorm / sqrt(var+epsilon) # shape (N, D)
+ }
+}
+
+init = function(int D)
+ return (matrix[double] gamma, matrix[double] beta,
+ matrix[double] ema_mean, matrix[double] ema_var) {
+ /*
+ * Initialize the parameters of this layer.
+ *
+ * Note: This is just a convenience function, and parameters
+ * may be initialized manually if needed.
+ *
+ * Inputs:
+ * - D: Dimensionality of the input features.
+ *
+ * Outputs:
+ * - gamma: Scale parameters, of shape (1, D).
+ * - beta: Shift parameters, of shape (1, D).
+ * - ema_mean: Exponential moving average of the mean, of
+ * shape (1, D).
+ * - ema_var: Exponential moving average of the variance, of
+ * shape (1, D).
+ */
+ gamma = matrix(1, rows=1, cols=D)
+ beta = matrix(0, rows=1, cols=D)
+ ema_mean = matrix(0, rows=1, cols=D)
+ ema_var = matrix(1, rows=1, cols=D)
+}
+
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/c944ad1d/scripts/staging/SystemML-NN/nn/layers/spatial_batch_norm.dml
----------------------------------------------------------------------
diff --git a/scripts/staging/SystemML-NN/nn/layers/spatial_batch_norm.dml b/scripts/staging/SystemML-NN/nn/layers/spatial_batch_norm.dml
new file mode 100644
index 0000000..53ca989
--- /dev/null
+++ b/scripts/staging/SystemML-NN/nn/layers/spatial_batch_norm.dml
@@ -0,0 +1,235 @@
+#-------------------------------------------------------------
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-------------------------------------------------------------
+
+/*
+ * Spatial batch normalization layer.
+ */
+source("nn/util.dml") as util
+
+forward = function(matrix[double] X, matrix[double] gamma, matrix[double] beta,
+ int C, int Hin, int Win, string mode,
+ matrix[double] ema_mean, matrix[double] ema_var,
+ double mu, double epsilon)
+ return (matrix[double] out, matrix[double] ema_mean_upd, matrix[double] ema_var_upd,
+ matrix[double] cache_mean, matrix[double] cache_var, matrix[double] cache_norm) {
+ /*
+ * Computes the forward pass for a spatial batch normalization layer.
+ *
+ * A spatial batch normalization layer uses the per-channel sample
+ * mean and per-channel uncorrected sample variance during training
+ * to normalize each channel of the input data. Additionally, it
+ * introduces learnable parameters (gamma, beta) to control the
+ * amount of normalization.
+ *
+ * y = ((x-mean) / sqrt(var+eps)) * gamma + beta
+ *
+ * This implementation maintains exponential moving averages of the
+ * mean and variance during training for use during testing.
+ *
+ * Reference:
+ * - Batch Normalization: Accelerating Deep Network Training by
+ * Reducing Internal Covariate Shift, S. Ioffe & C. Szegedy, 2015
+ * - https://arxiv.org/abs/1502.03167
+ *
+ * Inputs:
+ * - X: Input data matrix, of shape (N, C*Hin*Win).
+ * - gamma: Scale parameters, of shape (C, 1).
+ * - beta: Shift parameters, of shape (C, 1).
+ * - C: Number of input channels (dimensionality of input depth).
+ * - Hin: Input height.
+ * - Win: Input width.
+ * - mode: 'train' or 'test' to indicate if the model is currently
+ * being trained or tested. During training, the current batch
+ * mean and variance will be used to normalize the inputs, while
+ * during testing, the exponential average of the mean and
+ * variance over all previous batches will be used.
+ * - ema_mean: Exponential moving average of the mean, of
+ * shape (C, 1).
+ * - ema_var: Exponential moving average of the variance, of
+ * shape (C, 1).
+ * - mu: Momentum value for moving averages.
+ * Typical values are in the range of [0.9, 0.999].
+ * - epsilon: Smoothing term to avoid divide by zero errors.
+ * Typical values are in the range of [1e-5, 1e-3].
+ *
+ * Outputs:
+ * - out: Outputs, of shape (N, C*Hin*Win).
+ * - ema_mean_upd: Updated exponential moving average of the mean,
+ * of shape (C, 1).
+ * - ema_var_upd: Updated exponential moving average of the variance,
+ * of shape (C, 1).
+ * - cache_mean: Cache of the batch mean, of shape (C, 1).
+ * Note: This is used for performance during training.
+ * - cache_var: Cache of the batch variance, of shape (C, 1).
+ * Note: This is used for performance during training.
+ * - cache_norm: Cache of the normalized inputs, of
+ * shape (C, N*Hin*Win). Note: This is used for performance
+ * during training.
+ */
+ N = nrow(X)
+
+ if(mode == 'train') {
+ # Compute channel-wise mean and variance
+ # Since we don't have tensors, we will compute the means and variances in a piece-wise fashion.
+ # - mean of total group is mean of subgroup means
+ # - variance is the mean of the subgroup variances + the variance of the subgroup means
+ subgrp_means = matrix(colMeans(X), rows=C, cols=Hin*Win)
+ subgrp_vars = matrix(colVars(X) * ((N-1)/N), rows=C, cols=Hin*Win) # uncorrected variances
+ mean = rowMeans(subgrp_means) # shape (C, 1)
+ var = rowMeans(subgrp_vars) + rowVars(subgrp_means)*(((Hin*Win)-1)/(Hin*Win)) # shape (C, 1)
+ # Update moving averages
+ ema_mean_upd = mu*ema_mean + (1-mu)*mean
+ ema_var_upd = mu*ema_var + (1-mu)*var
+ }
+ else {
+ # Use moving averages of mean and variance during testing
+ mean = ema_mean
+ var = ema_var
+ ema_mean_upd = ema_mean
+ ema_var_upd = ema_var
+ }
+
+ # Normalize, shift, and scale
+ # norm = (X-mean)*(var+epsilon)^(-1/2)
+ # = (X-mean) / sqrt(var+epsilon)
+ centered = bias_add(X, -mean) # shape (N, C*Hin*Win)
+ norm = bias_multiply(centered, 1/sqrt(var+epsilon)) # shape (N, C*Hin*Win)
+ # out = norm*gamma + beta
+ scaled = bias_multiply(norm, gamma) # shape (N, C*Hin*Win)
+ out = bias_add(scaled, beta) # shape (N, C*Hin*Win)
+
+ # Save variable for backward pass
+ cache_mean = mean
+ cache_var = var
+ cache_norm = norm
+}
+
+backward = function(matrix[double] dout, matrix[double] out,
+ matrix[double] ema_mean_upd, matrix[double] ema_var_upd,
+ matrix[double] cache_mean, matrix[double] cache_var, matrix[double] cache_norm,
+ matrix[double] X, matrix[double] gamma, matrix[double] beta,
+ int C, int Hin, int Win, string mode,
+ matrix[double] ema_mean, matrix[double] ema_var,
+ double mu, double epsilon)
+ return (matrix[double] dX, matrix[double] dgamma, matrix[double] dbeta) {
+ /*
+ * Computes the backward pass for a spatial batch normalization layer.
+ *
+ * Inputs:
+ * - dout: Derivatives from upstream, of shape (N, C*Hin*Win).
+ * - out: Outputs from the forward pass, of shape (N, C*Hin*Win).
+ * - ema_mean_upd: Updated exponential moving average of the mean
+ * from the forward pass, of shape (C, 1).
+ * - ema_var_upd: Updated exponential moving average of the variance
+ * from the forward pass, of shape (C, 1).
+ * - cache_mean: Cache of the batch mean from the forward pass, of
+ * shape (C, 1). Note: This is used for performance during
+ * training.
+ * - cache_var: Cache of the batch variance from the forward pass,
+ * of shape (C, 1). Note: This is used for performance during
+ * training.
+ * - cache_norm: Cache of the normalized inputs from the forward
+ * pass, of shape (C, N*Hin*Win). Note: This is used for
+ * performance during training.
+ * - X: Input data matrix to the forward pass, of
+ * shape (N, C*Hin*Win).
+ * - gamma: Scale parameters, of shape (C, 1).
+ * - beta: Shift parameters, of shape (C, 1).
+ * - C: Number of input channels (dimensionality of input depth).
+ * - Hin: Input height.
+ * - Win: Input width.
+ * - mode: 'train' or 'test' to indicate if the model is currently
+ * being trained or tested. During training, the current batch
+ * mean and variance will be used to normalize the inputs, while
+ * during testing, the exponential average of the mean and
+ * variance over all previous batches will be used.
+ * - ema_mean: Exponential moving average of the mean, of
+ * shape (C, 1).
+ * - ema_var: Exponential moving average of the variance, of
+ * shape (C, 1).
+ * - mu: Momentum value for moving averages.
+ * Typical values are in the range of [0.9, 0.999].
+ * - epsilon: Smoothing term to avoid divide by zero errors.
+ * Typical values are in the range of [1e-5, 1e-3].
+ *
+ * Outputs:
+ * - dX: Gradient wrt X, of shape (N, C*Hin*Win).
+ * - dgamma: Gradient wrt W, of shape (C, 1).
+ * - dbeta: Gradient wrt b, of shape (C, 1).
+ *
+ */
+ N = nrow(X)
+ mean = cache_mean
+ var = cache_var
+ norm = cache_norm
+ centered = bias_add(X, -mean) # shape (N, C*Hin*Win)
+
+ if (mode == 'train') {
+ # Compute gradients during training
+ dgamma = util::channel_sums(norm*dout, C, Hin, Win) # shape (C, 1)
+ dbeta = util::channel_sums(dout, C, Hin, Win) # shape (C, 1)
+ dnorm = bias_multiply(dout, gamma) # shape (N, C*Hin*Win)
+ dvar = util::channel_sums((-1/2) * bias_multiply(centered, (var+epsilon)^(-3/2)) * dnorm,
+ C, Hin, Win) # shape (C, 1)
+ dmean_norm_branch = util::channel_sums(bias_multiply(dnorm, -1/sqrt(var+epsilon)), C, Hin, Win)
+ dmean_var_branch = util::channel_sums((-2/(N*Hin*Win)) * centered, C, Hin, Win)
+ dmean_var_branch = dmean_var_branch * dvar # we can't use a function within an expression yet
+ dmean = dmean_norm_branch + dmean_var_branch # shape (C, 1)
+ dX_norm_branch = bias_multiply(dnorm, 1/sqrt(var+epsilon))
+ dX_mean_branch = (1/(N*Hin*Win)) * bias_add(matrix(0, rows=1, cols=C*Hin*Win), dmean)
+ dX_var_branch = (2/(N*Hin*Win)) * bias_multiply(centered, dvar)
+ dX = dX_norm_branch + dX_mean_branch + dX_var_branch # shape (N, C*Hin*Win)
+ }
+ else {
+ # Compute gradients during testing
+ dgamma = util::channel_sums(norm*dout, C, Hin, Win) # shape (C, 1)
+ dbeta = util::channel_sums(dout, C, Hin, Win) # shape (C, 1)
+ dnorm = bias_multiply(dout, gamma) # shape (N, C*Hin*Win)
+ dX = bias_multiply(dnorm, 1/sqrt(var+epsilon)) # shape (N, C*Hin*Win)
+ }
+}
+
+init = function(int C)
+ return (matrix[double] gamma, matrix[double] beta,
+ matrix[double] ema_mean, matrix[double] ema_var) {
+ /*
+ * Initialize the parameters of this layer.
+ *
+ * Note: This is just a convenience function, and parameters
+ * may be initialized manually if needed.
+ *
+ * Inputs:
+ * - C: Number of input channels (dimensionality of input depth).
+ *
+ * Outputs:
+ * - gamma: Scale parameters, of shape (C, 1).
+ * - beta: Shift parameters, of shape (C, 1).
+ * - ema_mean: Exponential moving average of the mean, of
+ * shape (C, 1).
+ * - ema_var: Exponential moving average of the variance, of
+ * shape (C, 1).
+ */
+ gamma = matrix(1, rows=C, cols=1)
+ beta = matrix(0, rows=C, cols=1)
+ ema_mean = matrix(0, rows=C, cols=1)
+ ema_var = matrix(1, rows=C, cols=1)
+}
+
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/c944ad1d/scripts/staging/SystemML-NN/nn/test/grad_check.dml
----------------------------------------------------------------------
diff --git a/scripts/staging/SystemML-NN/nn/test/grad_check.dml b/scripts/staging/SystemML-NN/nn/test/grad_check.dml
index db923df..6b90d56 100644
--- a/scripts/staging/SystemML-NN/nn/test/grad_check.dml
+++ b/scripts/staging/SystemML-NN/nn/test/grad_check.dml
@@ -23,6 +23,7 @@
* Gradient checks for various architectures.
*/
source("nn/layers/affine.dml") as affine
+source("nn/layers/batch_norm.dml") as batch_norm
source("nn/layers/conv.dml") as conv
source("nn/layers/conv_builtin.dml") as conv_builtin
source("nn/layers/cross_entropy_loss.dml") as cross_entropy_loss
@@ -39,6 +40,7 @@ source("nn/layers/relu.dml") as relu
source("nn/layers/rnn.dml") as rnn
source("nn/layers/sigmoid.dml") as sigmoid
source("nn/layers/softmax.dml") as softmax
+source("nn/layers/spatial_batch_norm.dml") as spatial_batch_norm
source("nn/layers/tanh.dml") as tanh
source("nn/test/conv_simple.dml") as conv_simple
source("nn/test/max_pool_simple.dml") as max_pool_simple
@@ -161,6 +163,108 @@ affine = function() {
}
}
+batch_norm = function() {
+ /*
+ * Gradient check for the batch normalization layer.
+ */
+ print("Grad checking the batch normalization layer with L2 loss.")
+
+ # Generate data
+ N = 3 # num examples
+ D = 100 # num features
+ mu = 0.9 # momentum
+ eps = 1e-5 # epsilon
+ X = rand(rows=N, cols=D)
+ y = rand(rows=N, cols=D)
+ gamma = rand(rows=1, cols=D)
+ beta = rand(rows=1, cols=D)
+ ema_mean = rand(rows=1, cols=D)
+ ema_var = rand(rows=1, cols=D)
+ #[dummy, dummy, ema_mean, ema_var] = batch_norm::init(D)
+
+ # Check training & testing modes
+ for (i in 1:2) {
+ if (i == 1)
+ mode = 'train'
+ else
+ mode = 'test'
+ print(" - Grad checking the '"+mode+"' mode.")
+
+ # Compute analytical gradients of loss wrt parameters
+ [out, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ dout = l2_loss::backward(out, y)
+ [dX, dgamma, dbeta] = batch_norm::backward(dout, out, ema_mean_upd, ema_var_upd,
+ cache_mean, cache_var, cache_norm,
+ X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+
+ # Grad check
+ h = 1e-5
+ print(" - Grad checking X.")
+ for (i in 1:nrow(X)) {
+ for (j in 1:ncol(X)) {
+ # Compute numerical derivative
+ old = as.scalar(X[i,j])
+ X[i,j] = old - h
+ [outmh, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ lossmh = l2_loss::forward(outmh, y)
+ X[i,j] = old + h
+ [outph, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ lossph = l2_loss::forward(outph, y)
+ X[i,j] = old # reset
+ dX_num = (lossph - lossmh) / (2 * h) # numerical derivative
+
+ # Check error
+ rel_error = check_rel_error(as.scalar(dX[i,j]), dX_num, lossph, lossmh)
+ }
+ }
+
+ print(" - Grad checking gamma.")
+ for (i in 1:nrow(gamma)) {
+ for (j in 1:ncol(gamma)) {
+ # Compute numerical derivative
+ old = as.scalar(gamma[i,j])
+ gamma[i,j] = old - h
+ [outmh, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ lossmh = l2_loss::forward(outmh, y)
+ gamma[i,j] = old + h
+ [outph, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ lossph = l2_loss::forward(outph, y)
+ gamma[i,j] = old # reset
+ dgamma_num = (lossph - lossmh) / (2 * h) # numerical derivative
+
+ # Check error
+ rel_error = check_rel_error(as.scalar(dgamma[i,j]), dgamma_num, lossph, lossmh)
+ }
+ }
+
+ print(" - Grad checking beta.")
+ for (i in 1:nrow(beta)) {
+ for (j in 1:ncol(beta)) {
+ # Compute numerical derivative
+ old = as.scalar(beta[i,j])
+ beta[i,j] = old - h
+ [outmh, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ lossmh = l2_loss::forward(outmh, y)
+ beta[i,j] = old + h
+ [outph, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+ lossph = l2_loss::forward(outph, y)
+ beta[i,j] = old # reset
+ dbeta_num = (lossph - lossmh) / (2 * h) # numerical derivative
+
+ # Check error
+ rel_error = check_rel_error(as.scalar(dbeta[i,j]), dbeta_num, lossph, lossmh)
+ }
+ }
+ }
+}
+
conv = function() {
/*
* Gradient check for the convolutional layer using `im2col`.
@@ -1203,6 +1307,118 @@ softmax = function() {
}
}
+spatial_batch_norm = function() {
+ /*
+ * Gradient check for the spatial batch normalization layer.
+ */
+ print("Grad checking the spatial batch normalization layer with L2 loss.")
+
+ # Generate data
+ N = 3 # num examples
+ N = 2 # num examples
+ C = 2 # num channels
+ Hin = 5 # input height
+ Win = 5 # input width
+ mu = 0.9 # momentum
+ eps = 1e-5 # epsilon
+ X = rand(rows=N, cols=C*Hin*Win)
+ y = rand(rows=N, cols=C*Hin*Win)
+ gamma = rand(rows=C, cols=1)
+ beta = rand(rows=C, cols=1)
+ ema_mean = rand(rows=C, cols=1)
+ ema_var = rand(rows=C, cols=1)
+ #[dummy, dummy, ema_mean, ema_var] = spatial_batch_norm::init(C)
+
+ # Check training & testing modes
+ for (i in 1:2) {
+ if (i == 1)
+ mode = 'train'
+ else
+ mode = 'test'
+ print(" - Grad checking the '"+mode+"' mode.")
+
+ # Compute analytical gradients of loss wrt parameters
+ [out, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode, ema_mean, ema_var, mu, eps)
+ dout = l2_loss::backward(out, y)
+ [dX, dgamma, dbeta] = spatial_batch_norm::backward(dout, out, ema_mean_upd, ema_var_upd,
+ cache_mean, cache_var, cache_norm,
+ X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+
+ # Grad check
+ h = 1e-5
+ print(" - Grad checking X.")
+ for (i in 1:nrow(X)) {
+ for (j in 1:ncol(X)) {
+ # Compute numerical derivative
+ old = as.scalar(X[i,j])
+ X[i,j] = old - h
+ [outmh, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+ lossmh = l2_loss::forward(outmh, y)
+ X[i,j] = old + h
+ [outph, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+ lossph = l2_loss::forward(outph, y)
+ X[i,j] = old # reset
+ dX_num = (lossph - lossmh) / (2 * h) # numerical derivative
+
+ # Check error
+ rel_error = check_rel_error(as.scalar(dX[i,j]), dX_num, lossph, lossmh)
+ }
+ }
+
+ print(" - Grad checking gamma.")
+ for (i in 1:nrow(gamma)) {
+ for (j in 1:ncol(gamma)) {
+ # Compute numerical derivative
+ old = as.scalar(gamma[i,j])
+ gamma[i,j] = old - h
+ [outmh, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+ lossmh = l2_loss::forward(outmh, y)
+ gamma[i,j] = old + h
+ [outph, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+ lossph = l2_loss::forward(outph, y)
+ gamma[i,j] = old # reset
+ dgamma_num = (lossph - lossmh) / (2 * h) # numerical derivative
+
+ # Check error
+ rel_error = check_rel_error(as.scalar(dgamma[i,j]), dgamma_num, lossph, lossmh)
+ }
+ }
+
+ print(" - Grad checking beta.")
+ for (i in 1:nrow(beta)) {
+ for (j in 1:ncol(beta)) {
+ # Compute numerical derivative
+ old = as.scalar(beta[i,j])
+ beta[i,j] = old - h
+ [outmh, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+ lossmh = l2_loss::forward(outmh, y)
+ beta[i,j] = old + h
+ [outph, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode,
+ ema_mean, ema_var, mu, eps)
+ lossph = l2_loss::forward(outph, y)
+ beta[i,j] = old # reset
+ dbeta_num = (lossph - lossmh) / (2 * h) # numerical derivative
+
+ # Check error
+ rel_error = check_rel_error(as.scalar(dbeta[i,j]), dbeta_num, lossph, lossmh)
+ }
+ }
+ }
+}
+
tanh = function() {
/*
* Gradient check for the hyperbolic tangent (tanh) nonlinearity layer.
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/c944ad1d/scripts/staging/SystemML-NN/nn/test/test.dml
----------------------------------------------------------------------
diff --git a/scripts/staging/SystemML-NN/nn/test/test.dml b/scripts/staging/SystemML-NN/nn/test/test.dml
index dd24a55..b25fae2 100644
--- a/scripts/staging/SystemML-NN/nn/test/test.dml
+++ b/scripts/staging/SystemML-NN/nn/test/test.dml
@@ -22,16 +22,51 @@
/*
* Various tests, not including gradient checks.
*/
+source("nn/layers/batch_norm.dml") as batch_norm
source("nn/layers/conv.dml") as conv
source("nn/layers/conv_builtin.dml") as conv_builtin
source("nn/layers/cross_entropy_loss.dml") as cross_entropy_loss
source("nn/layers/max_pool.dml") as max_pool
source("nn/layers/max_pool_builtin.dml") as max_pool_builtin
+source("nn/layers/spatial_batch_norm.dml") as spatial_batch_norm
source("nn/layers/tanh.dml") as tanh
source("nn/test/conv_simple.dml") as conv_simple
source("nn/test/max_pool_simple.dml") as max_pool_simple
source("nn/util.dml") as util
+batch_norm = function() {
+ /*
+ * Test for the `batch_norm` function.
+ */
+ print("Testing the batch_norm function.")
+
+ # Generate data
+ N = 4 # Number of examples
+ D = 4 # Number of features
+ mode = 'train' # execution mode
+ mu = 0.9 # momentum of moving averages
+ eps = 1e-5 # smoothing term
+ X = matrix(seq(1,16), rows=N, cols=D)
+
+ # Create layer
+ [gamma, beta, ema_mean, ema_var] = batch_norm::init(D)
+
+ # Forward
+ [out, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ batch_norm::forward(X, gamma, beta, mode, ema_mean, ema_var, mu, eps)
+
+ # Equivalency check
+ target = matrix("-1.34160721 -1.34160721 -1.34160733 -1.34160709
+ -0.44720244 -0.44720244 -0.44720244 -0.44720232
+ 0.44720244 0.44720232 0.44720244 0.44720244
+ 1.34160733 1.34160721 1.34160733 1.34160733", rows=1, cols=N*D)
+ out = matrix(out, rows=1, cols=N*D)
+ for (i in 1:length(out)) {
+ rel_error = util::check_rel_error(as.scalar(out[1,i]),
+ as.scalar(target[1,i]), 1e-3, 1e-4)
+ }
+}
+
conv = function() {
/*
* Test for the `conv` functions.
@@ -189,8 +224,8 @@ max_pool = function() {
for (padh in 0:3) {
for (padw in 0:3) {
print(" - Testing w/ padh="+padh+" & padw="+padw+".")
- if (1==1) {} # force correct printing
- print(" - Testing forward")
+ #if (1==1) {} # force correct printing
+ #print(" - Testing forward")
[out, Hout, Wout] = max_pool::forward(X, C, Hin, Win, Hf, Wf, stride, stride, padh, padw)
[out_simple, Hout_simple, Wout_simple] = max_pool_simple::forward(X, C, Hin, Win, Hf, Wf,
stride, stride, padh, padw)
@@ -209,7 +244,7 @@ max_pool = function() {
as.scalar(out_builtin[1,i]), 1e-10, 1e-12)
}
- print(" - Testing backward")
+ #print(" - Testing backward")
dout = rand(rows=N, cols=C*Hout*Wout, pdf="normal")
dX = max_pool::backward(dout, Hout, Wout, X, C, Hin, Win, Hf, Wf, stride, stride, padh, padw)
dX_simple = max_pool_simple::backward(dout, Hout_simple, Wout_simple, X, C, Hin, Win, Hf, Wf,
@@ -388,6 +423,97 @@ max_pool = function() {
tmp = util::check_all_equal(out_builtin, target)
}
+spatial_batch_norm = function() {
+ /*
+ * Test for the `spatial_batch_norm` function.
+ */
+ print("Testing the spatial_batch_norm function.")
+
+ # Generate data
+ N = 2 # Number of examples
+ C = 3 # num channels
+ Hin = 4 # input height
+ Win = 5 # input width
+ mode = 'train' # execution mode
+ mu = 0.9 # momentum of moving averages
+ eps = 1e-5 # smoothing term
+ X = matrix("70 29 23 55 72
+ 42 98 68 48 39
+ 34 73 44 6 40
+ 74 18 18 53 53
+
+ 63 85 72 61 72
+ 32 36 23 29 63
+ 9 43 43 49 43
+ 31 43 89 94 50
+
+ 62 12 32 41 87
+ 25 48 99 52 61
+ 12 83 60 55 34
+ 30 42 68 88 51
+
+
+ 67 59 62 67 84
+ 8 76 24 19 57
+ 10 89 63 72 2
+ 59 56 16 15 70
+
+ 32 69 55 39 93
+ 84 36 4 30 40
+ 70 100 36 76 59
+ 69 15 40 24 34
+
+ 51 67 11 13 32
+ 66 85 55 85 38
+ 32 35 17 83 34
+ 55 58 52 0 99", rows=N, cols=C*Hin*Win)
+
+ # Create layer
+ [gamma, beta, ema_mean, ema_var] = spatial_batch_norm::init(C)
+
+ # Forward
+ [out, ema_mean_upd, ema_var_upd, cache_mean, cache_var, cache_norm] =
+ spatial_batch_norm::forward(X, gamma, beta, C, Hin, Win, mode, ema_mean, ema_var, mu, eps)
+
+ # Equivalency check
+ target = matrix("0.86215019 -0.76679718 -1.00517964 0.26619387 0.94161105
+ -0.25030172 1.97460198 0.78268933 -0.01191914 -0.36949289
+ -0.56814504 0.98134136 -0.17084086 -1.68059683 -0.32976246
+ 1.02107191 -1.20383179 -1.20383179 0.18673301 0.18673301
+
+ 0.50426388 1.41921711 0.87856293 0.42108631 0.87856293
+ -0.78498828 -0.61863315 -1.15928721 -0.90975463 0.50426388
+ -1.74153018 -0.32751167 -0.32751167 -0.07797909 -0.32751167
+ -0.82657707 -0.32751167 1.58557224 1.79351616 -0.0363903
+
+ 0.4607178 -1.49978399 -0.71558321 -0.36269283 1.44096887
+ -0.99005347 -0.08822262 1.91148913 0.06861746 0.42150795
+ -1.49978399 1.28412855 0.38229787 0.18624771 -0.63716316
+ -0.79400325 -0.32348287 0.69597805 1.48017895 0.0294075
+
+
+ 0.74295878 0.42511559 0.54430676 0.74295878 1.41837597
+ -1.60113597 1.10053277 -0.96544927 -1.16410136 0.34565473
+ -1.52167511 1.61702824 0.5840373 0.94161105 -1.83951855
+ 0.42511559 0.30592418 -1.28329265 -1.32302308 0.86215019
+
+ -0.78498828 0.75379658 0.17155361 -0.4938668 1.75192738
+ 1.37762833 -0.61863315 -1.9494741 -0.86816585 -0.45227802
+ 0.79538536 2.04304862 -0.61863315 1.04491806 0.33790874
+ 0.75379658 -1.49199748 -0.45227802 -1.11769855 -0.70181072
+
+ 0.0294075 0.65676796 -1.53899395 -1.46057391 -0.71558321
+ 0.61755812 1.36254871 0.18624771 1.36254871 -0.48032296
+ -0.71558321 -0.59795308 -1.30373383 1.28412855 -0.63716316
+ 0.18624771 0.30387771 0.06861746 -1.97030437 1.91148913", rows=1,
+ cols=N*C*Hin*Win)
+ out = matrix(out, rows=1, cols=N*C*Hin*Win)
+ for (i in 1:length(out)) {
+ rel_error = util::check_rel_error(as.scalar(out[1,i]),
+ as.scalar(target[1,i]), 1e-3, 1e-4)
+ }
+}
+
tanh = function() {
/*
* Test for the `tanh` forward function.
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/c944ad1d/scripts/staging/SystemML-NN/nn/test/tests.dml
----------------------------------------------------------------------
diff --git a/scripts/staging/SystemML-NN/nn/test/tests.dml b/scripts/staging/SystemML-NN/nn/test/tests.dml
index fd897ef..86bb77b 100644
--- a/scripts/staging/SystemML-NN/nn/test/tests.dml
+++ b/scripts/staging/SystemML-NN/nn/test/tests.dml
@@ -29,11 +29,15 @@ print("")
print("Starting grad checks.")
print("---")
+# Loss functions
tmp = grad_check::cross_entropy_loss()
tmp = grad_check::l1_loss()
tmp = grad_check::l2_loss()
tmp = grad_check::log_loss()
+
+# Other layers
tmp = grad_check::affine()
+tmp = grad_check::batch_norm()
tmp = grad_check::conv_simple()
tmp = grad_check::conv()
tmp = grad_check::conv_builtin()
@@ -48,7 +52,10 @@ tmp = grad_check::relu()
tmp = grad_check::rnn()
tmp = grad_check::sigmoid()
tmp = grad_check::softmax()
+tmp = grad_check::spatial_batch_norm()
tmp = grad_check::tanh()
+
+# Example model
tmp = grad_check::two_layer_affine_l2_net()
print("---")
@@ -62,11 +69,13 @@ print("")
print("Starting other tests.")
print("---")
+tmp = test::batch_norm()
tmp = test::im2col()
tmp = test::padding()
tmp = test::conv()
tmp = test::cross_entropy_loss()
tmp = test::max_pool()
+tmp = test::spatial_batch_norm()
tmp = test::tanh()
print("---")
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/c944ad1d/scripts/staging/SystemML-NN/nn/util.dml
----------------------------------------------------------------------
diff --git a/scripts/staging/SystemML-NN/nn/util.dml b/scripts/staging/SystemML-NN/nn/util.dml
index fdf0f76..dd0ac19 100644
--- a/scripts/staging/SystemML-NN/nn/util.dml
+++ b/scripts/staging/SystemML-NN/nn/util.dml
@@ -111,6 +111,25 @@ check_rel_error = function(double x1, double x2, double thresh_error, double thr
}
}
+channel_sums = function(matrix[double] X, int C, int Hin, int Win)
+ return (matrix[double] out) {
+ /*
+ * Computes a channel-wise summation over a 4D input.
+ *
+ * Inputs:
+ * - X: Input data matrix, of shape (N, C*Hin*Win).
+ * - C: Number of input channels (dimensionality of input depth).
+ * - Hin: Input height.
+ * - Win: Input width.
+ *
+ * Outputs:
+ * - out: Outputs, of shape (C, 1).
+ */
+ # Here we sum each column, reshape to (C, Hin*Win), and sum each row to result in the summation
+ # for each channel.
+ out = rowSums(matrix(colSums(X), rows=C, cols=Hin*Win)) # shape (C, 1)
+}
+
im2col = function(matrix[double] img, int Hin, int Win, int Hf, int Wf, int strideh, int stridew)
return (matrix[double] img_cols) {
/*