[GitHub] mkolod commented on issue #6946: Optimized sequence reverse operator

[gi...@git.apache.org]

mkolod commented on issue #6946: Optimized sequence reverse operator
URL: https://github.com/apache/incubator-mxnet/pull/6946#issuecomment-318766929
 
 
   @fhieber @piiswrong 
   
   I tried to repro the slowdown and was unable to. I compared the top of tree to the TOT with the commit in questions (a8804a) unapplied, on the following machine:
   
   - Core i7-5930K CPU @ 3.50GHz (6 cores)
   - 32 GB RAM
   - Tesla P100 (Pascal)
   
   While I don't see a slowdown, the speed-up is very minor, after the various refactorings done in the process of the code review for this PR. Initially, it was a difference between 20% of the execution time (prior to PR version 1) to 0.5% of the execution time. Unfortunately, since padding cannot be inverted and requires branching, and due to some other changes since the first commit, the gain is now 2% at best.
   
   The Sockeye parameters I used were as follows (on WMT15 DE-EN dataset):
   
   ```
   python3 -m sockeye.train \
           --source data/train2.de \
           --target data/train2.en \
           --validation-source data/valid2.de \
           --validation-target data/valid2.en \
           --output models \
           --num-words 50000 \
           --rnn-num-layers 2 \
           --rnn-num-hidden 512 \
           --num-embed-source 512 \
           --num-embed-target 512 \
           --max-seq-len 50 \
           --batch-size 64 \
           --bucket-width 10 \
           --checkpoint-frequency 20000 \
           --max-num-checkpoint-not-improved 8 \
           --dropout 0.3 \
           --optimizer adam \
           --clip-gradient 1.0 \
           --learning-rate-scheduler-type plateau-reduce \
           --learning-rate-reduce-factor 0.5 \
           --learning-rate-reduce-num-not-improved 3 \
           --attention-type dot \
           --learning-rate-half-life 10 \
           --monitor-bleu 10000 \
           --seed 42 \
           --use-fused-rnn \
           --device-ids 1 \
           --max-updates 2000
   ```
   
   | MxNet version        | 1k batches           | 2k batches  |
   | ------------- |:-------------:| -----:|
   |  TOT      | 225s | 420s |
   | TOT with commit a8804a unapplied       | 221s     | 419s  |
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services