You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/10/09 13:59:10 UTC
[GitHub] xfwu opened a new issue #8190: why cudnn_lstm_bucketing.py got so low ppl
xfwu opened a new issue #8190: why cudnn_lstm_bucketing.py got so low ppl
URL: https://github.com/apache/incubator-mxnet/issues/8190
## Environment info
Operating System: Centos
Compiler:
Package used (Python/R/Scala/Julia):
MXNet version: 0.11
Python version and distribution:
2.7.13
## Error Message:
I run the example/rnn/cudnn_lstm_bucketing.py with
python cudnn_lstm_bucketing.py --gpus 0,1 --num-epochs 500 --num-embed 500 --num-hidden 1200 --bidirectional False --batch-size 246 --dropout 0.7 --lr 1.0 --model-prefix train.unidir.nostrip
I got
.......
2017-10-09 13:26:20,817 Epoch[19] Validation-perplexity=13.750214
2017-10-09 13:26:35,388 Epoch[20] Batch [50] Speed: 854.62 samples/sec perplexity=16.620398
2017-10-09 13:26:49,914 Epoch[20] Batch [100] Speed: 846.79 samples/sec perplexity=17.194405
2017-10-09 13:27:03,739 Epoch[20] Batch [150] Speed: 889.70 samples/sec perplexity=16.455508
2017-10-09 13:27:08,195 Epoch[20] Train-perplexity=16.163508
2017-10-09 13:27:08,196 Epoch[20] Time cost=47.315
2017-10-09 13:27:09,060 Saved checkpoint to "train.unidir.nostrip-0021.params"
2017-10-09 13:27:10,253 Epoch[20] Validation-perplexity=11.188143
2017-10-09 13:27:24,578 Epoch[21] Batch [50] Speed: 876.79 samples/sec perplexity=14.935809
2017-10-09 13:27:38,669 Epoch[21] Batch [100] Speed: 872.88 samples/sec perplexity=15.025512
2017-10-09 13:27:52,974 Epoch[21] Batch [150] Speed: 859.88 samples/sec perplexity=15.260417
2017-10-09 13:27:57,688 Epoch[21] Train-perplexity=15.116437
2017-10-09 13:27:57,688 Epoch[21] Time cost=47.391
2017-10-09 13:27:58,618 Saved checkpoint to "train.unidir.nostrip-0022.params"
2017-10-09 13:27:59,806 Epoch[21] Validation-perplexity=10.612535
2017-10-09 13:28:14,102 Epoch[22] Batch [50] Speed: 875.98 samples/sec perplexity=13.628904
2017-10-09 13:28:27,892 Epoch[22] Batch [100] Speed: 891.96 samples/sec perplexity=13.669974
2017-10-09 13:28:41,738 Epoch[22] Batch [150] Speed: 888.38 samples/sec perplexity=14.028721
2017-10-09 13:28:47,157 Epoch[22] Train-perplexity=14.196291
2017-10-09 13:28:47,157 Epoch[22] Time cost=47.271
2017-10-09 13:28:48,097 Saved checkpoint to "train.unidir.nostrip-0023.params"
2017-10-09 13:28:49,277 Epoch[22] Validation-perplexity=32.945784
2017-10-09 13:29:03,641 Epoch[23] Batch [50] Speed: 865.63 samples/sec perplexity=16.444668
2017-10-09 13:29:17,674 Epoch[23] Batch [100] Speed: 876.53 samples/sec perplexity=14.599101
2017-10-09 13:29:31,426 Epoch[23] Batch [150] Speed: 894.46 samples/sec perplexity=13.958345
2017-10-09 13:29:36,668 Epoch[23] Train-perplexity=13.931759
2017-10-09 13:29:36,668 Epoch[23] Time cost=47.349
2017-10-09 13:29:37,562 Saved checkpoint to "train.unidir.nostrip-0024.params"
2017-10-09 13:29:38,748 Epoch[23] Validation-perplexity=11.139995
..........
which I think there is no way PTB can achieve such low ppl.
## What have you tried to solve it?
I tried to trace down the error, I think the predicted probability is somehow wrong.
I trained with another toy corpus which I use <s> and </s> to wrap every sentence. After the training, when I test on a sentence "<s> <unk> </s>"
on python -m ptb, In the metrics.py, I got
-> pred = ndarray.pick(pred, label.astype(dtype='int32'), axis=self.axis)
(Pdb) p label
[ 73. 6. -1.]
(Pdb) p pred[0][73]
[ 0.99999988]
<NDArray 1 @cpu(0)>
(Pdb) p pred[0][2463]
[ 1.23341704e-09]
<NDArray 1 @cpu(0)>
( the 73 means <unk> and 6 means </s> )
**whereas for another sentence "<s> trunks </s>" ( 2463 is the word trunks)
(Pdb) p label
[ 2.46300000e+03 6.00000000e+00 -1.00000000e+00]
(Pdb) p pred[0][73]
[ 0.00913999]
Pdb) p pred[0][2463]
[ 7.78183894e-05]
I think lstm should never give different output given only input "<s>",,, there seemed something wrong.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services