[GitHub] ThomasDelteil commented on a change in pull request #10350: Fix Gluon Language Model Example

[GitBox <gi...@apache.org>]

ThomasDelteil commented on a change in pull request #10350: Fix Gluon Language Model Example
URL: https://github.com/apache/incubator-mxnet/pull/10350#discussion_r178434307
 
 

 ##########
 File path: example/gluon/word_language_model/train.py
 ##########
 @@ -159,19 +160,19 @@ def train():
             hidden = detach(hidden)
             with autograd.record():
                 output, hidden = model(data, hidden)
+                # Here L is a vector of size batch_size * bptt size
                 L = loss(output, target)
+                L = L / (args.bptt * args.batch_size)
                 L.backward()
 
             grads = [p.grad(context) for p in model.collect_params().values()]
-            # Here gradient is for the whole batch.
-            # So we multiply max_norm by batch_size and bptt size to balance it.
-            gluon.utils.clip_global_norm(grads, args.clip * args.bptt * args.batch_size)
+            gluon.utils.clip_global_norm(grads, args.clip)
 
-            trainer.step(args.batch_size)
+            trainer.step(1)
 
 Review comment:
   Why do you change the step to 1?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services