You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/08/02 18:43:03 UTC
[GitHub] [incubator-mxnet] ChengXianbing opened a new issue #15738: AMP
results in higher loss
ChengXianbing opened a new issue #15738: AMP results in higher loss
URL: https://github.com/apache/incubator-mxnet/issues/15738
After AMP training for this GluonNLP training model:
https://github.com/dmlc/gluon-nlp/compare/master...ChengXianbing:master
we found that the loss is higher than training without enabling AMP
(We also enable AMP training in our own language model, we found that not only the loss is
higher, but also the training throughput decreases from 30445.52 samples/s to 25603.86 samples/s).
Could you please help out?
```
$ python word_language_model.py --model gru --emsize 64 --nhid 128 --lr 1.0 --epochs 3 --bptt 10 --tied --nlayers 1 --test_mode --gpu 0
[Epoch 0] throughput 25758.38 samples/s
[Epoch 0] time cost 0.44s, valid loss 10.12, valid ppl 24864.71,lr 1.00
[Epoch 0] test loss 10.15, test ppl 25633.89
[Epoch 1] throughput 71637.50 samples/s
[Epoch 1] time cost 0.36s, valid loss 9.06, valid ppl 8631.06,lr 1.00
[Epoch 1] test loss 9.21, test ppl 10037.64
[Epoch 2] throughput 66607.71 samples/s
[Epoch 2] time cost 0.30s, valid loss 8.27, valid ppl 3919.10,lr 1.00
[Epoch 2] test loss 8.45, test ppl 4673.45
Total training throughput 14113.57 samples/s
Best validation loss 8.27, val ppl 3919.10
Best test loss 8.45, test ppl 4673.45
Total time cost 1.82s
$ python word_language_model.py --model gru --emsize 64 --nhid 128 --lr 1.0 --epochs 3 --bptt 10 --tied --nlayers 1 --test_mode --amp_training --gpu 0
[Epoch 0] throughput 36236.71 samples/s
[Epoch 0] time cost 0.44s, valid loss 10.41, valid ppl 33323.06,lr 1.00
[Epoch 0] test loss 10.41, test ppl 33251.08
[Epoch 1] throughput 64450.66 samples/s
[Epoch 1] time cost 0.32s, valid loss 10.41, valid ppl 33322.87,lr 1.00
[Epoch 1] test loss 10.41, test ppl 33250.94
[Epoch 2] throughput 63184.95 samples/s
[Epoch 2] time cost 0.33s, valid loss 10.41, valid ppl 33322.68,lr 1.00
[Epoch 2] test loss 10.41, test ppl 33250.76
Total training throughput 14169.86 samples/s
Best validation loss 10.41, val ppl 33322.68
Best test loss 10.41, test ppl 33250.76
Total time cost 1.82s
```
```
$ nvidia-smi -q
==============NVSMI LOG==============
Timestamp : Fri Aug 2 18:38:22 2019
Driver Version : 418.67
CUDA Version : 10.1
Attached GPUs : 1
GPU 00000000:00:1E.0
Product Name : Tesla V100-SXM2-16GB
Product Brand : Tesla
Display Mode : Enabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0322917091773
GPU UUID : GPU-e9a16ab2-2c86-8a0b-1126-8511d8165cd5
Minor Number : 0
VBIOS Version : 88.00.4F.00.09
MultiGPU Board : No
Board ID : 0x1e
GPU Part Number : 900-2G503-0000-000
Inforom Version
Image Version : G503.0201.00.03
OEM Object : 1.1
ECC Object : 5.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : Pass-Through
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x00
Device : 0x1E
Domain : 0x0000
Device Id : 0x1DB110DE
Bus Id : 00000000:00:1E.0
Sub System Id : 0x121210DE
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 0 KB/s
Rx Throughput : 0 KB/s
Fan Speed : N/A
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
FB Memory Usage
Total : 16130 MiB
Used : 0 MiB
Free : 16130 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Default
Utilization
Gpu : 4 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : 0
Total : 0
Aggregate
Single Bit
Device Memory : 5
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : 5
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : 0
Total : 0
Retired Pages
Single Bit ECC : 1
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 50 C
GPU Shutdown Temp : 90 C
GPU Slowdown Temp : 87 C
GPU Max Operating Temp : 83 C
Memory Current Temp : 45 C
Memory Max Operating Temp : 85 C
Power Readings
Power Management : Supported
Power Draw : 42.75 W
Power Limit : 300.00 W
Default Power Limit : 300.00 W
Enforced Power Limit : 300.00 W
Min Power Limit : 150.00 W
Max Power Limit : 300.00 W
Clocks
Graphics : 1312 MHz
SM : 1312 MHz
Memory : 877 MHz
Video : 1177 MHz
Applications Clocks
Graphics : 1312 MHz
Memory : 877 MHz
Default Applications Clocks
Graphics : 1312 MHz
Memory : 877 MHz
Max Clocks
Graphics : 1530 MHz
SM : 1530 MHz
Memory : 877 MHz
Video : 1372 MHz
Max Customer Boost Clocks
Graphics : 1530 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
```
mxnet version: ```mxnet-cu101 1.5.0b20190711```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services