[GitHub] absalama opened a new issue #11200: Training Imagenet on AlexNet failed with corrupted JPEG format images

[GitBox <gi...@apache.org>]

absalama opened a new issue #11200: Training Imagenet on AlexNet failed with corrupted JPEG format images 
URL: https://github.com/apache/incubator-mxnet/issues/11200
 
 
   Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
   
   For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io 
   
   ## Description
   I am training imagenet using alexnet. I converted the images to .rec  files  that format Mxnet requires using tools/im2rec. However, the training process failed due to error in the jpeg format (After one hour from training). 
   
   ## Environment info (Required)
   
   
   
   Package used (Python/R/Scala/Julia):
   (I'm using ...)
   
   For Scala user, please provide:
   1. Java version: (`java -version`)
   2. Maven version: (`mvn -version`)
   3. Scala runtime if applicable: (`scala -version`)
   
   For R user, please provide R `sessionInfo()`:
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio):
   
   MXNet commit hash:
   (Paste the output of `git rev-parse HEAD` here.)
   
   Build config:
   (Paste the content of config.mk, or the build command.)
   
   ## Error Message:
   INFO:root:Epoch[0] Batch [1780]	Speed: 34.83 samples/sec	accuracy=0.003125
   Corrupt JPEG data: premature end of data segment
   terminate called after throwing an instance of 'dmlc::Error'
     what():  [17:20:52] src/recordio.cc:117: Check failed: p[0] == RecordIOWriter::kMagic 
   
   Stack trace returned 6 entries:
   [bt] (0) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x30cbe2) [0x2b9eb9b4abe2]
   [bt] (1) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2f9e845) [0x2b9ebc7dc845]
   [bt] (2) /work/projects/Project00755/envs/mxnet_gpu/lib/python3.5/site-packages/mxnet/libmxnet.so(+0x2a34e98) [0x2b9ebc272e98]
   [bt] (3) /shared/apps/gcc/4.9.4/lib64/libgomp.so.1(+0xed36) [0x2b9ef5042d36]
   [bt] (4) /lib64/libpthread.so.0(+0x7e25) [0x2b9e8961fe25]
   [bt] (5) /lib64/libc.so.6(clone+0x6d) [0x2b9e8a03534d]
   
   
   /opt/slurm/current/var/spool/job7708145/slurm_script: line 28: 31099 Aborted                 (core dumped)
   ## Minimum reproducible example
   (If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)
   
   ## Steps to reproduce
   (Paste the commands you ran that produced the error.)
   
   python3 train_imagenet.py --gpus 0 --data-nthreads 8 --network alexnet --batch-size 128 --model mxnet_alexnet_single_gpu --lr 0.01 --optimizer sgd --mom 0.9 --wd 0.0005 --image-shape 3,227,227 --num-examples 1281144 --data-train "${IMAGENET_ROOT}/record_io/train/train.rec" --data-val "${IMAGENET_ROOT}/record_io/val/value.rec" --data-train-idx "${IMAGENET_ROOT}/record_io/train/train.idx" --data-val-idx "${IMAGENET_ROOT}/record_io/val/value.idx"
   
   ## What have you tried to solve it?
   I deleted the corrupted images from my raw dataset (was approx 24 images with mcyk) and then I run the im2rec tool to convert the images to .rec. However that does not help
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services