You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by gi...@git.apache.org on 2017/10/08 14:56:40 UTC
[GitHub] qinhui99 opened a new issue #8178: BIig bug in training kpi(the model is overfitting,but the kpi say everything is ok)
qinhui99 opened a new issue #8178: BIig bug in training kpi(the model is overfitting,but the kpi say everything is ok)
URL: https://github.com/apache/incubator-mxnet/issues/8178
## Environment info
Operating System: win10
Compiler:
Package used (Python/R/Scala/Julia): python2.7 and 3.5
MXNet version: 0.11 GPU version
## Error Message:
I tried to challenge the kaggle project (https://www.kaggle.com/c/state-farm-distracted-driver-detection) with mxnet. And I used the pre-trained model named resnet50 to train the model. At the beginning, everything seems ok. The training kpi( acc and log loss) were very good. I show the log infos as follows:
Read train data time: 75.85 seconds
('x.shape', (15696L, 3L, 64L, 64L))
ready to train!
[21:44:51] G:\deeplearn\mxnet\src\nnvm\legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[21:44:51] G:\deeplearn\mxnet\src\nnvm\legacy_json_util.cc:198: Symbol successfully upgraded!
[21:44:53] g:\deeplearn\mxnet\src\operator\./cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
INFO:root:Epoch[0] Train-accuracy=0.123438
INFO:root:Epoch[0] Train-log_loss=2.323805
INFO:root:Epoch[0] Time cost=8.827
INFO:root:Epoch[0] Validation-accuracy=0.131676
INFO:root:Epoch[0] Validation-log_loss=2.319754
INFO:root:Epoch[1] Train-accuracy=0.169313
INFO:root:Epoch[1] Train-log_loss=2.244676
INFO:root:Epoch[1] Time cost=7.668
INFO:root:Epoch[1] Validation-accuracy=0.195312
INFO:root:Epoch[1] Validation-log_loss=2.215750
INFO:root:Epoch[2] Train-accuracy=0.233187
INFO:root:Epoch[2] Train-log_loss=2.150264
INFO:root:Epoch[2] Time cost=7.682
INFO:root:Epoch[2] Validation-accuracy=0.250568
INFO:root:Epoch[2] Validation-log_loss=2.121630
INFO:root:Epoch[3] Train-accuracy=0.332750
INFO:root:Epoch[3] Train-log_loss=2.020984
INFO:root:Epoch[3] Time cost=7.695
INFO:root:Epoch[3] Validation-accuracy=0.374716
INFO:root:Epoch[3] Validation-log_loss=1.960711
INFO:root:Epoch[4] Train-accuracy=0.454125
INFO:root:Epoch[4] Train-log_loss=1.837035
INFO:root:Epoch[4] Time cost=7.703
INFO:root:Saved checkpoint to "mxnet_resnet50_argument-0005.params"
INFO:root:Epoch[4] Validation-accuracy=0.500852
INFO:root:Epoch[4] Validation-log_loss=1.765853
INFO:root:Epoch[5] Train-accuracy=0.599688
INFO:root:Epoch[5] Train-log_loss=1.589363
INFO:root:Epoch[5] Time cost=7.714
INFO:root:Epoch[5] Validation-accuracy=0.614062
INFO:root:Epoch[5] Validation-log_loss=1.500510
INFO:root:Epoch[6] Train-accuracy=0.729500
INFO:root:Epoch[6] Train-log_loss=1.294998
INFO:root:Epoch[6] Time cost=7.720
INFO:root:Epoch[6] Validation-accuracy=0.735369
INFO:root:Epoch[6] Validation-log_loss=1.185761
INFO:root:Epoch[7] Train-accuracy=0.828063
INFO:root:Epoch[7] Train-log_loss=0.969563
INFO:root:Epoch[7] Time cost=7.742
INFO:root:Epoch[7] Validation-accuracy=0.816193
INFO:root:Epoch[7] Validation-log_loss=0.933975
INFO:root:Epoch[8] Train-accuracy=0.901937
INFO:root:Epoch[8] Train-log_loss=0.687442
INFO:root:Epoch[8] Time cost=7.735
INFO:root:Epoch[8] Validation-accuracy=0.865341
INFO:root:Epoch[8] Validation-log_loss=0.730559
INFO:root:Epoch[9] Train-accuracy=0.953500
INFO:root:Epoch[9] Train-log_loss=0.462237
INFO:root:Epoch[9] Time cost=7.749
INFO:root:Saved checkpoint to "mxnet_resnet50_argument-0010.params"
INFO:root:Epoch[9] Validation-accuracy=0.895739
INFO:root:Epoch[9] Validation-log_loss=0.597818
INFO:root:Epoch[10] Train-accuracy=0.982563
INFO:root:Epoch[10] Train-log_loss=0.297865
INFO:root:Epoch[10] Time cost=7.746
INFO:root:Epoch[10] Validation-accuracy=0.909233
INFO:root:Epoch[10] Validation-log_loss=0.500578
INFO:root:Epoch[11] Train-accuracy=0.996563
INFO:root:Epoch[11] Train-log_loss=0.188816
INFO:root:Epoch[11] Time cost=7.760
INFO:root:Epoch[11] Validation-accuracy=0.917330
INFO:root:Epoch[11] Validation-log_loss=0.440542
INFO:root:Epoch[12] Train-accuracy=0.999500
INFO:root:Epoch[12] Train-log_loss=0.126055
INFO:root:Epoch[12] Time cost=7.761
INFO:root:Epoch[12] Validation-accuracy=0.922727
INFO:root:Epoch[12] Validation-log_loss=0.402706
INFO:root:Epoch[13] Train-accuracy=0.999938
INFO:root:Epoch[13] Train-log_loss=0.089558
INFO:root:Epoch[13] Time cost=7.776
INFO:root:Epoch[13] Validation-accuracy=0.925426
INFO:root:Epoch[13] Validation-log_loss=0.373640
INFO:root:Epoch[14] Train-accuracy=1.000000
INFO:root:Epoch[14] Train-log_loss=0.067061
INFO:root:Epoch[14] Time cost=7.791
INFO:root:Saved checkpoint to "mxnet_resnet50_argument-0015.params"
INFO:root:Epoch[14] Validation-accuracy=0.926278
INFO:root:Epoch[14] Validation-log_loss=0.353491
----------------------------------------------------------------------------------
As you can see, in the last turn of training ,we got the best local score :
Validation-accuracy=0.926278
Validation-log_loss=0.353491
Are they very good?
But, when I used the model to predict the test data and submited into Kaggle, I got the very bad score.
public score:2.08153 ,private score 2.44852. This kaggle project used the log loss as the metric. why there are too much big gap between the mxnet local kpi and kaggle ? Firstly, I guessed the labels were wrong in test data. But I tried to check many times and found the labels were right. Secondly, I rewrite my codes in other models ( mlp , cnn ) and train them. They are all ok in training kpi. But when I submited into Kaggle, the score were still very bad ( the scores are between 2.1 and 6.5 ).
So I guess it is overfitting for my local model.
In fact , I did many times for the same test in Keras( tensorflow backend). I used the resnet50 model to train the same pictures. In training , the Keras kpi can tell me the model is overfitting after several epoches. If I got a 0.68 log loss in keras, then I will get the closer score( 0.7) in Kaggle. It is reliable.
In the end, I must say that the training kpi in mxnet are very unreliable. My model is too overfitting ,but the training kpi say everything is ok and just go on. Is it a big bug ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services