You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/02/06 17:11:59 UTC

[GitHub] nazikus opened a new issue #9715: IndexError in labels when size of training dataset is not multiple of batch size

nazikus opened a new issue #9715: IndexError in labels when size of training dataset is not multiple of batch size
URL: https://github.com/apache/incubator-mxnet/issues/9715
 
 
   ## Description
   
   IndexError is thrown during training if training dataset size if not multiple of batch size.
   
   ## Long description
   
   I have a simple transfer learning algorithm based on [mxnet tutorial](https://mxnet.incubator.apache.org/how_to/finetune.html).
   However, I had a nasty exception during training that occurs every time on the last batch of the first epoch. Empirically I have figure out if you manually ensure that dataset size (number of records in .lst file) is multiple of batch size,  than this exception is gone.
   
   **Is it expected behavior, or is proper validation missing?**
   
   
   ## Environment info (Required)
   
       python 2.7.12
       mxnet-cu91mkl 1.0.0.post4
       CUDA 9.1.85
   
   
   ## Error Message:
   
   IndexError example during training:
   
       2018-02-06 16:24:39,605 - Epoch[1] Batch [200]	Speed: 37.22 samples/sec	cross-entropy=3.293303	accuracy=0.192188
       2018-02-06 16:25:13,382 - Epoch[1] Batch [220]	Speed: 37.90 samples/sec	cross-entropy=3.286735	accuracy=0.190625
       Traceback (most recent call last):
         File "train_multilabel.py", line 204, in <module>
           train_model(args)
         File "train_multilabel.py", line 135, in train_model
           num_epoch          = params.num_epoch,
         File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/module/base_module.py", line 496, in fit
           self.update_metric(eval_metric, data_batch.label)
         File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/module/module.py", line 749, in update_metric
           self._exec_group.update_metric(eval_metric, labels)
         File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/module/executor_group.py", line 616, in update_metric
           eval_metric.update_dict(labels_, preds)
         File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/metric.py", line 281, in update_dict
           metric.update_dict(labels, preds)
         File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/metric.py", line 109, in update_dict
           self.update(label, pred)
         File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/metric.py", line 924, in update
           prob = pred[numpy.arange(label.shape[0]), numpy.int64(label)]
       IndexError: index -9223372036854775808 is out of bounds for axis 1 with size 52
       
   
   
   ## Steps to reproduce
   
   simply create ImageIter object with dataset size (number of lines in .lst file) not divisible by batch size, and iterate over it, print label values:
   
       kv = mx.kvstore.create('device')
       
       train_data = mx.image.ImageIter(
           batch_size   = 64,
           data_shape   = (3, 224, 224),
           path_imglist = 'dataset/train_data.lst',
           path_root    = 'dataset/images/,
           part_index   = kv.rank,
           num_parts    = kv.num_workers,
           shuffle      = True,
           data_name    = 'data',
           label_name   = 'softmax_label',
       )
   
       for i, batch in enumerate(train_data):
           print("batch idx {:3d}\n{}\n".format(i, batch.label[0].asnumpy().tolist()))
   
   For example, dataset size 15226, batch size 64, number of batches 238 (rounded). 
   
   In my case, some remaining label values in the last batch are non-integers:
   
       ...
       batch index   236
       [24.0, 13.0, 29.0, 21.0, 48.0, 44.0, 22.0, 47.0, 36.0, 9.0, 47.0, 43.0, 33.0, 3.0, 25.0, 34.0, 47.0, 1.0, 34.0, 40.0, 11.0, 10.0, 43.0, 3.0, 43.0, 27.0, 39.0, 39.0, 13.0, 48.0, 28.0, 42.0, 24.0, 39.0, 31.0, 45.0, 51.0, 6.0, 1.0, 48.0, 17.0, 42.0, 23.0, 9.0, 27.0, 39.0, 19.0, 36.0, 14.0, 10.0, 26.0, 37.0, 42.0, 7.0, 47.0, 29.0, 37.0, 6.0, 9.0, 9.0, 39.0, 5.0, 11.0, 22.0]
       batch index   237
       [5.0, 50.0, 11.0, 12.0, 50.0, 10.0, 35.0, 31.0, 11.0, 13.0, 2.0, 26.0, 51.0, 6.0, 48.0, 37.0, 25.0, 24.0, 14.0, 20.0, 44.0, 40.0, 21.0, 45.0, 23.0, 18.0, 10.0, 15.0, 21.0, 7.0, 33.0, 32.0, 50.0, 44.0, 10.0, 22.0, 7.0, 9.0, 3.0, 7.0, 49.0, 47.0, 49.0, 26.0, 0.0, 23.0, 2.0, 1.0, 27.0, 0.0, 13.0, 18.0, 38.0, 27.0, 50.0, 18.0, 46.0, 5.0, 7.315163621112927e-37, 0.0, 7.315152859140721e-37, 0.0, 7.315142097168515e-37, 0.0]
   
   
   ## What have you tried to solve it?
   manually remove several lines from `trainining_data.lst` file, so total number of training dataset size (eg 15168) is divisible by batch size (64).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services