[2/2] incubator-systemml git commit: [SYSTEMML-618] Stabilizing the `two_layer_affine_l2_net` grad check.

[du...@apache.org]

[SYSTEMML-618] Stabilizing the `two_layer_affine_l2_net` grad check.

The example `two_layer_affine_l2_net` network uses a ReLU nonlinearity,
could result in a false-negative in which the test fails due to a kink
being crossed in the ReLU nonlinearity.  This occurs when the tests,
f(x-h) and f(x+h), end up on opposite sides of the zero threshold of
max(0, fx).  This change aims to stabilize the test, while still
allowing for the usage of the ReLU nonlinearity.  Although we have
stabilized it, we will still remove it from automated testing to avoid
false failures.  In the future, we can explicitly check for this
scenario and rerun the test automatically.  For manual testing, simply
rerun the tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/42ebdef5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/42ebdef5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/42ebdef5

Branch: refs/heads/master
Commit: 42ebdef5d38ad421e2f478eefb8bea535d402309
Parents: fb53098
Author: Mike Dusenberry <mw...@us.ibm.com>
Authored: Fri Apr 28 17:51:13 2017 -0700
Committer: Mike Dusenberry <mw...@us.ibm.com>
Committed: Fri Apr 28 17:54:14 2017 -0700

----------------------------------------------------------------------
 scripts/nn/test/grad_check.dml |  5 +++--
 scripts/nn/test/run_tests.dml  | 10 +++++++++-
 2 files changed, 12 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/42ebdef5/scripts/nn/test/grad_check.dml
----------------------------------------------------------------------
diff --git a/scripts/nn/test/grad_check.dml b/scripts/nn/test/grad_check.dml
index f3bc9a7..e767eee 100644
--- a/scripts/nn/test/grad_check.dml
+++ b/scripts/nn/test/grad_check.dml
@@ -1612,11 +1612,12 @@ two_layer_affine_l2_net = function() {
   M = 10 # number of hidden neurons
   [W1, b1] = affine::init(D, M)
   [W2, b2] = affine::init(M, yD)
+  W2 = W2 / sqrt(2)  # different initialization, since being fed into l2 loss, instead of relu
 
   # Optimize for short "burn-in" time to move to characteristic
   # mode of operation and unmask any real issues.
   print(" - Burn-in:")
-  lr = 0.0001
+  lr = 0.01
   decay = 0.99
   for(i in 1:5) {
     # Compute forward and backward passes of net
@@ -1635,7 +1636,7 @@ two_layer_affine_l2_net = function() {
   [pred, loss, dX, dW1, db1, dW2, db2] = two_layer_affine_l2_net_run(X, y, W1, b1, W2, b2)
 
   # Grad check
-  h = 1e-5
+  h = 1e-6
   print(" - Grad checking X.")
   for (i in 1:2) {
     for (j in 1:ncol(X)) {

http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/42ebdef5/scripts/nn/test/run_tests.dml
----------------------------------------------------------------------
diff --git a/scripts/nn/test/run_tests.dml b/scripts/nn/test/run_tests.dml
index d8173a9..b061d26 100644
--- a/scripts/nn/test/run_tests.dml
+++ b/scripts/nn/test/run_tests.dml
@@ -60,7 +60,15 @@ grad_check::tanh()
 print("")
 
 # Example model
-grad_check::two_layer_affine_l2_net()
+# NOTE: This could result in a false-negative in which the test fails
+# due to a kink being crossed in the ReLU nonlinearity.  This occurs
+# when the tests, f(x-h) and f(x+h), end up on opposite sides of the
+# zero threshold of max(0, fx).  Although we have stabilized it, we
+# will still remove it here to avoid false failures during automated
+# testing. In the future, we can explicitly check for this scenario
+# and rerun the test automatically.  For manual testing, simply
+# rerun the tests.
+#grad_check::two_layer_affine_l2_net()
 print("")
 
 print("---")