You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/10/09 13:59:10 UTC

[GitHub] xfwu opened a new issue #8190: why cudnn_lstm_bucketing.py got so low ppl

xfwu opened a new issue #8190: why cudnn_lstm_bucketing.py got so low ppl
URL: https://github.com/apache/incubator-mxnet/issues/8190
 
 
   ## Environment info
   Operating System: Centos
   
   Compiler:
   
   Package used (Python/R/Scala/Julia):
   
   MXNet version: 0.11
   
   Python version and distribution:
   2.7.13
   
   ## Error Message:
   I run the example/rnn/cudnn_lstm_bucketing.py with
   
   python cudnn_lstm_bucketing.py   --gpus 0,1 --num-epochs 500 --num-embed 500 --num-hidden 1200 --bidirectional False --batch-size 246 --dropout 0.7  --lr 1.0  --model-prefix train.unidir.nostrip
   
   I got 
   .......
   2017-10-09 13:26:20,817 Epoch[19] Validation-perplexity=13.750214
   2017-10-09 13:26:35,388 Epoch[20] Batch [50]    Speed: 854.62 samples/sec   perplexity=16.620398
   2017-10-09 13:26:49,914 Epoch[20] Batch [100]   Speed: 846.79 samples/sec   perplexity=17.194405
   2017-10-09 13:27:03,739 Epoch[20] Batch [150]   Speed: 889.70 samples/sec   perplexity=16.455508
   2017-10-09 13:27:08,195 Epoch[20] Train-perplexity=16.163508
   2017-10-09 13:27:08,196 Epoch[20] Time cost=47.315
   2017-10-09 13:27:09,060 Saved checkpoint to "train.unidir.nostrip-0021.params"
   2017-10-09 13:27:10,253 Epoch[20] Validation-perplexity=11.188143
   2017-10-09 13:27:24,578 Epoch[21] Batch [50]    Speed: 876.79 samples/sec   perplexity=14.935809
   2017-10-09 13:27:38,669 Epoch[21] Batch [100]   Speed: 872.88 samples/sec   perplexity=15.025512
   2017-10-09 13:27:52,974 Epoch[21] Batch [150]   Speed: 859.88 samples/sec   perplexity=15.260417
   2017-10-09 13:27:57,688 Epoch[21] Train-perplexity=15.116437
   2017-10-09 13:27:57,688 Epoch[21] Time cost=47.391
   2017-10-09 13:27:58,618 Saved checkpoint to "train.unidir.nostrip-0022.params"
   2017-10-09 13:27:59,806 Epoch[21] Validation-perplexity=10.612535
   2017-10-09 13:28:14,102 Epoch[22] Batch [50]    Speed: 875.98 samples/sec   perplexity=13.628904
   2017-10-09 13:28:27,892 Epoch[22] Batch [100]   Speed: 891.96 samples/sec   perplexity=13.669974
   2017-10-09 13:28:41,738 Epoch[22] Batch [150]   Speed: 888.38 samples/sec   perplexity=14.028721
   2017-10-09 13:28:47,157 Epoch[22] Train-perplexity=14.196291
   2017-10-09 13:28:47,157 Epoch[22] Time cost=47.271
   2017-10-09 13:28:48,097 Saved checkpoint to "train.unidir.nostrip-0023.params"
   2017-10-09 13:28:49,277 Epoch[22] Validation-perplexity=32.945784
   2017-10-09 13:29:03,641 Epoch[23] Batch [50]    Speed: 865.63 samples/sec   perplexity=16.444668
   2017-10-09 13:29:17,674 Epoch[23] Batch [100]   Speed: 876.53 samples/sec   perplexity=14.599101
   2017-10-09 13:29:31,426 Epoch[23] Batch [150]   Speed: 894.46 samples/sec   perplexity=13.958345
   2017-10-09 13:29:36,668 Epoch[23] Train-perplexity=13.931759
   2017-10-09 13:29:36,668 Epoch[23] Time cost=47.349
   2017-10-09 13:29:37,562 Saved checkpoint to "train.unidir.nostrip-0024.params"
   2017-10-09 13:29:38,748 Epoch[23] Validation-perplexity=11.139995
   ..........
   which I think there is no way PTB can achieve such low ppl.
   
   ## What have you tried to solve it?
   I tried to trace down the error, I think the predicted probability is somehow wrong.
   
   I trained with another toy corpus which I use <s> and </s> to wrap every sentence. After the training, when I test on a sentence "<s> <unk> </s>"
   on python -m ptb, In the metrics.py, I got 
   -> pred = ndarray.pick(pred, label.astype(dtype='int32'), axis=self.axis)
   (Pdb) p label
   [ 73.   6.  -1.]
   (Pdb) p pred[0][73]
   [ 0.99999988]
   <NDArray 1 @cpu(0)>
   (Pdb) p pred[0][2463]
   [  1.23341704e-09]
   <NDArray 1 @cpu(0)>
   ( the 73 means <unk> and 6 means </s> )
   **whereas for another sentence "<s> trunks </s>" ( 2463 is the word trunks)
   (Pdb) p label
   [  2.46300000e+03   6.00000000e+00  -1.00000000e+00]
   (Pdb) p pred[0][73]
   [ 0.00913999]
   Pdb) p pred[0][2463]
   [  7.78183894e-05]
   
   I think lstm should never give different output given only input "<s>",,, there seemed something wrong.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services