[GitHub] [incubator-mxnet] onomatet commented on issue #17207: LR schedulers do not work in R

[GitBox <gi...@apache.org>]

onomatet commented on issue #17207: LR schedulers do not work in R
URL: https://github.com/apache/incubator-mxnet/issues/17207#issuecomment-570788495
 
 
   ## Minimal example
   
   I hope it will be of some help. Modified after the tutorial.
   
   ### Code
   ```
   data(BostonHousing, package="mlbench")
   
   train.ind = seq(1, 506, 3)
   train.x = data.matrix(BostonHousing[train.ind, -14])
   train.y = BostonHousing[train.ind, 14]
   
   data <- mx.symbol.Variable("data")
   fc1 <- mx.symbol.FullyConnected(data, num_hidden=1)
   lro <- mx.symbol.LinearRegressionOutput(fc1)
   
   cat("Without scheduler\n")
   mx.set.seed(0)
   model1 <- mx.model.FeedForward.create(lro, X=train.x, y=train.y,
       ctx=mx.cpu(),     num.round=20, array.batch.size=20,
       learning.rate=2e-6, momentum=0.9,  eval.metric=mx.metric.rmse)
   
   cat("\nWith scheduler\n")
   lr_scheduler <- mx.lr_scheduler.FactorScheduler(
       step = 5 * ceiling(length(train.y)/20), factor = 0.1,
       stop_factor_lr = 1e-10)
   mx.set.seed(0)
   model2 <- mx.model.FeedForward.create(lro, X=train.x, y=train.y,
       ctx=mx.cpu(),     num.round=20, array.batch.size=20,
       learning.rate=2e-6, momentum=0.9,  eval.metric=mx.metric.rmse,
       lr_scheduler = lr_scheduler)
   ```
   
   ### Output
   
   ```
   Without scheduler
   Start training with 1 devices
   [1] Train-rmse=18.0516391330295
   [2] Train-rmse=13.9097522099813
   [3] Train-rmse=10.666042221917
   [4] Train-rmse=10.0117386711968
   [5] Train-rmse=9.59162097507053
   [6] Train-rmse=9.80227173699273
   [7] Train-rmse=9.56405830383301
   [8] Train-rmse=9.39033126831055
   [9] Train-rmse=9.33245415157742
   [10] Train-rmse=9.31073543760512
   [11] Train-rmse=9.27919896443685
   [12] Train-rmse=9.24656009674072
   [13] Train-rmse=9.206680615743
   [14] Train-rmse=9.17186906602648
   [15] Train-rmse=9.14681609471639
   [16] Train-rmse=9.12289328045315
   [17] Train-rmse=9.09742567274306
   [18] Train-rmse=9.0733421113756
   [19] Train-rmse=9.05105861028036
   [20] Train-rmse=9.02993933359782
   
   With scheduler
   Start training with 1 devices
   [1] Train-rmse=18.0516391330295
   [2] Train-rmse=13.9097522099813
   [3] Train-rmse=10.666042221917
   [4] Train-rmse=10.0117386711968
   [5] Train-rmse=9.59162097507053
   Update[46]: learning rate is changed to 2e-07
   [6] Train-rmse=9.80227173699273
   [7] Train-rmse=9.56405830383301
   [8] Train-rmse=9.39033126831055
   [9] Train-rmse=9.33245415157742
   [10] Train-rmse=9.31073543760512
   Update[91]: learning rate is changed to 2e-08
   [11] Train-rmse=9.27919896443685
   [12] Train-rmse=9.24656009674072
   [13] Train-rmse=9.206680615743
   [14] Train-rmse=9.17186906602648
   [15] Train-rmse=9.14681609471639
   Update[136]: learning rate is changed to 2e-09
   [16] Train-rmse=9.12289328045315
   [17] Train-rmse=9.09742567274306
   [18] Train-rmse=9.0733421113756
   [19] Train-rmse=9.05105861028036
   [20] Train-rmse=9.02993933359782
   Warning messages:
   1: In mx.model.select.layout.train(X, y) :
     Auto detect layout of input matrix, use rowmajor..
   
   2: In mx.model.select.layout.train(X, y) :
     Auto detect layout of input matrix, use rowmajor..
   ```
   
   Train-rmse is the same in every iteration regardless of the scheduler used.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services