[GitHub] thomasmooon opened a new issue #7595: MXNetR: How to corrupt input layer? (

[gi...@git.apache.org]

thomasmooon opened a new issue #7595: MXNetR: How to corrupt input layer? (
URL: https://github.com/apache/incubator-mxnet/issues/7595
 
 
   I've asked a similar, but not the same question in issue #7490 which was misleading.
   The difference is: The intention is not to dropout input units, rather corrupt the input by setting values to zero. I don't know how to implement it in a net using symbolic language.
   
   How can one implement the "corrupting procedure" into the "net"?
   
   # corrupting procedure
   p <- 0.25
   set.seed(1)
   x <- mx.nd.array(matrix(runif(25,1,20),nrow=5))
   mask <- mx.nd.uniform(0,1,dim(x)) 
   mask <- mx.nd.broadcast.lesser(xCoru,mx.nd.array(1-p))
   
   ### original input
   x
   ```
             [,1]      [,2]      [,3]      [,4]
   [1,]  6.044664  4.831957 12.953167 14.053434
   [2,]  8.070354 18.069405  2.173939  8.297971
   [3,] 11.884213 18.948830  4.913517 15.626987
   [4,] 18.255947 13.555158  4.354578 10.456285
   ```
   ### corruption mask
   mask
      ```
     [,1] [,2] [,3] [,4]
   [1,]    0    1    0    1
   [2,]    0    0    0    0
   [3,]    1    0    1    1
   [4,]    0    1    1    0
   ```
   
   ### apply corruption
   x*mask
   ```
            [,1]      [,2]     [,3]     [,4]
   [1,]  0.00000  4.831957 0.000000 14.05343
   [2,]  0.00000  0.000000 0.000000  0.00000
   [3,] 11.88421  0.000000 4.913517 15.62699
   [4,]  0.00000 13.555158 4.354578  0.0000
   ```
   
   # net
   
   ```
   create_drop_net <- function(num_hidden=10, p = 0.2) {
     data <- mx.symbol.Variable("data")
   ```
   corrupt input, but how??
   This won't work:
   ```
   mask<- mx.nd.uniform(0,1,dim(data))
   mask <- mx.nd.broadcast.lesser(xCoru,mx.nd.array(1-p))
   data <- data*mask
   ```
   
   ```
     label <- mx.symbol.Variable('label')
     fc1 <- mx.symbol.FullyConnected(data, name = "fc1", num_hidden = num_hidden)
     act1 <- mx.symbol.Activation(fc1, name = "relu1", act_type = "relu")
     fc2 <- mx.symbol.FullyConnected(act1, name = "fc2", num_hidden = num_hidden)
     act2 <- mx.symbol.Activation(fc2, name = "relu2", act_type = "relu")
     fc3 <- mx.symbol.FullyConnected(act2, name = "fc3", num_hidden = num_hidden)
     act3 <- mx.symbol.Activation(fc3, name = "relu3", act_type = "relu")
     recon <- mx.symbol.FullyConnected(act3, name = "reconstruction", num_hidden = nrow(train))
     net <- mx.symbol.LinearRegressionOutput(data = recon, label = label)
     return(net)
   }
   ```
   
   the rest of the code (data iterator and training)
   ```
   # data iterator
   batchIter <- 128
   trainIter <- mx.io.arrayiter(data = train, label = train, batch.size = batchIter)
   valIter <- mx.io.arrayiter(data = test, label = test, batch.size = batchIter)
   
   # train
   mx.set.seed(0)
   model <- mx.model.FeedForward.create(
     create_drop_net(num_hidden = 10),
     X = trainIter, 
     eval.data = valIter,
     ctx = device,
     num.round = 10,
     array.batch.size = length(device)*128,
     learning.rate = 0.1,
     momentum = 0.9,
     eval.metric = mx.metric.rmse, 
     array.layout = "colmajor",
     initializer = mx.init.uniform(0.1),
     epoch.end.callback = mx.callback.log.train.metric(10)
   )
   ```
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services