[GitHub] juliusshufan opened a new issue #9199: Convergence issue on training ResNet-50 on "tinyImageNet" dataset

[GitBox <gi...@apache.org>]

juliusshufan opened a new issue #9199: Convergence issue on training ResNet-50 on "tinyImageNet" dataset 
URL: https://github.com/apache/incubator-mxnet/issues/9199
 
 
   ## Description
   I am using the MXNET with MKL2017 enabled on a Xeon-based machine.
   I installed the MXNET from source code and compilation is okay. I am trying to train a ResNet-50 model on the official "tinyImageNet" dataset. I did notice the training trended to converge with epochs/iteration ongoing (accuracy increasing to almost 0.9), with the following two excepetions:
   1. At the end of each epochs, sometimes the printed training accuracy is nan;
   2. The validation accuracy can not exceed 0.37, even the training accuracy of each iteration increasing.
   
   ## Environment info (Required)
   OS: CentOS 7.2 
   GCC: 4.8.5
   Python 2.7.5
   dataset link: http://cs231n.stanford.edu/tiny-imagenet-200.zip 
   
   ## Data prepocess
   The dataset contains 100, 200 samples for training and 10, 000 samples for validation, all selected from ImageNet and resized to 64*64. The original training set already organized as the "sub-foldering" per category, but the validation set was NOT organized as the "sub-foldering" per category.
    
   I reorganized the validation set as "sub-foldering" per category, according to the txt-based annotation file enclosed with the dataset, and convert the dataset to rec file.
   
   The training script following the MxNet official examples for image-classification and using the symbols coming with the MxNet official examples. (That is "example/image-classification/symbols")
   
   May I have any clues?
   
   Thanks.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services