[GitHub] [systemds] tobiasrieger commented on a change in pull request #1026: [MINOR] Added leaky relu layer + grad check and updated 2NN example

[GitBox <gi...@apache.org>]

tobiasrieger commented on a change in pull request #1026:
URL: https://github.com/apache/systemds/pull/1026#discussion_r474814290



##########
File path: scripts/nn/examples/mnist_2NN.dml
##########
@@ -20,159 +20,161 @@
 #-------------------------------------------------------------
 
 /*
- * MNIST 2NN Relu Example
- */
+* MNIST 2NN Leaky Relu Example
+*/
 
 # Imports
 source("nn/layers/affine.dml") as affine
 source("nn/layers/cross_entropy_loss.dml") as cross_entropy_loss
-source("nn/layers/relu.dml") as relu
+source("nn/layers/leaky_relu.dml") as leaky_relu
 source("nn/layers/softmax.dml") as softmax
 source("nn/optim/sgd_nesterov.dml") as sgd_nesterov
 
 train = function(matrix[double] X, matrix[double] Y,
-                 matrix[double] X_val, matrix[double] Y_val,
-                 int epochs)
-    return (matrix[double] W_1, matrix[double] b_1,
-            matrix[double] W_2, matrix[double] b_2,
-            matrix[double] W_3, matrix[double] b_3) {
-  /*
-   * Trains a 2NN relu softmax classifier.
-   *
-   * The input matrix, X, has N examples, each with D features.
-   * The targets, Y, have K classes, and are one-hot encoded.
-   *
-   * Inputs:
-   *  - X: Input data matrix, of shape (N, D).
-   *  - Y: Target matrix, of shape (N, K).
-   *  - X_val: Input validation data matrix, of shape (N, C*Hin*Win).
-   *  - Y_val: Target validation matrix, of shape (N, K).
-   *  - epochs: Total number of full training loops over the full data set.
-   *
-   * Outputs:
-   *  - W: Weights (parameters) matrix, of shape (D, M, 3).
-   *  - b: Biases vector, of shape (1, M, 3).
-   */
-  N = nrow(X)  # num examples
-  D = ncol(X)  # num features
-  K = ncol(Y)  # num classes
+matrix[double] X_val, matrix[double] Y_val,
+int epochs)
+return (matrix[double] W_1, matrix[double] b_1,
+matrix[double] W_2, matrix[double] b_2,
+matrix[double] W_3, matrix[double] b_3) {
 
-  # Create the network:
-  # input -> 200 neuron affine -> relu -> 200 neuron affine -> relu -> K neurons affine -> softmax
-  [W_1, b_1] = affine::init(D, 200)
-  [W_2, b_2] = affine::init(200, 200)
-  [W_3, b_3] = affine::init(200, K)
+/*
+* Trains a 2 hidden layer leaky relu softmax classifier.
+*
+* The input matrix, X, has N examples, each with D features.
+* The targets, Y, have K classes, and are one-hot encoded.
+*
+* Inputs:
+*  - X: Input data matrix, of shape (N, D).
+*  - Y: Target matrix, of shape (N, K).
+*  - X_val: Input validation data matrix, of shape (N, C*Hin*Win).
+*  - Y_val: Target validation matrix, of shape (N, K).
+*  - epochs: Total number of full training loops over the full data set.
+*
+* Outputs:
+*  - W: Weights (parameters) matrix, of shape (D, M, 3).
+*  - b: Biases vector, of shape (1, M, 3).
+*/
+

Review comment:
       Sorry, I missed this. It is fixed now. The remaining formatting changes are planned.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org