You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/02/06 17:11:59 UTC
[GitHub] nazikus opened a new issue #9715: IndexError in labels when size of training dataset is not multiple of batch size
nazikus opened a new issue #9715: IndexError in labels when size of training dataset is not multiple of batch size
URL: https://github.com/apache/incubator-mxnet/issues/9715
## Description
IndexError is thrown during training if training dataset size if not multiple of batch size.
## Long description
I have a simple transfer learning algorithm based on [mxnet tutorial](https://mxnet.incubator.apache.org/how_to/finetune.html).
However, I had a nasty exception during training that occurs every time on the last batch of the first epoch. Empirically I have figure out if you manually ensure that dataset size (number of records in .lst file) is multiple of batch size, than this exception is gone.
**Is it expected behavior, or is proper validation missing?**
## Environment info (Required)
python 2.7.12
mxnet-cu91mkl 1.0.0.post4
CUDA 9.1.85
## Error Message:
IndexError example during training:
2018-02-06 16:24:39,605 - Epoch[1] Batch [200] Speed: 37.22 samples/sec cross-entropy=3.293303 accuracy=0.192188
2018-02-06 16:25:13,382 - Epoch[1] Batch [220] Speed: 37.90 samples/sec cross-entropy=3.286735 accuracy=0.190625
Traceback (most recent call last):
File "train_multilabel.py", line 204, in <module>
train_model(args)
File "train_multilabel.py", line 135, in train_model
num_epoch = params.num_epoch,
File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/module/base_module.py", line 496, in fit
self.update_metric(eval_metric, data_batch.label)
File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/module/module.py", line 749, in update_metric
self._exec_group.update_metric(eval_metric, labels)
File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/module/executor_group.py", line 616, in update_metric
eval_metric.update_dict(labels_, preds)
File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/metric.py", line 281, in update_dict
metric.update_dict(labels, preds)
File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/metric.py", line 109, in update_dict
self.update(label, pred)
File "/home/otkach/.virtualenvs/mxnet-cu91mkl/local/lib/python2.7/site-packages/mxnet/metric.py", line 924, in update
prob = pred[numpy.arange(label.shape[0]), numpy.int64(label)]
IndexError: index -9223372036854775808 is out of bounds for axis 1 with size 52
## Steps to reproduce
simply create ImageIter object with dataset size (number of lines in .lst file) not divisible by batch size, and iterate over it, print label values:
kv = mx.kvstore.create('device')
train_data = mx.image.ImageIter(
batch_size = 64,
data_shape = (3, 224, 224),
path_imglist = 'dataset/train_data.lst',
path_root = 'dataset/images/,
part_index = kv.rank,
num_parts = kv.num_workers,
shuffle = True,
data_name = 'data',
label_name = 'softmax_label',
)
for i, batch in enumerate(train_data):
print("batch idx {:3d}\n{}\n".format(i, batch.label[0].asnumpy().tolist()))
For example, dataset size 15226, batch size 64, number of batches 238 (rounded).
In my case, some remaining label values in the last batch are non-integers:
...
batch index 236
[24.0, 13.0, 29.0, 21.0, 48.0, 44.0, 22.0, 47.0, 36.0, 9.0, 47.0, 43.0, 33.0, 3.0, 25.0, 34.0, 47.0, 1.0, 34.0, 40.0, 11.0, 10.0, 43.0, 3.0, 43.0, 27.0, 39.0, 39.0, 13.0, 48.0, 28.0, 42.0, 24.0, 39.0, 31.0, 45.0, 51.0, 6.0, 1.0, 48.0, 17.0, 42.0, 23.0, 9.0, 27.0, 39.0, 19.0, 36.0, 14.0, 10.0, 26.0, 37.0, 42.0, 7.0, 47.0, 29.0, 37.0, 6.0, 9.0, 9.0, 39.0, 5.0, 11.0, 22.0]
batch index 237
[5.0, 50.0, 11.0, 12.0, 50.0, 10.0, 35.0, 31.0, 11.0, 13.0, 2.0, 26.0, 51.0, 6.0, 48.0, 37.0, 25.0, 24.0, 14.0, 20.0, 44.0, 40.0, 21.0, 45.0, 23.0, 18.0, 10.0, 15.0, 21.0, 7.0, 33.0, 32.0, 50.0, 44.0, 10.0, 22.0, 7.0, 9.0, 3.0, 7.0, 49.0, 47.0, 49.0, 26.0, 0.0, 23.0, 2.0, 1.0, 27.0, 0.0, 13.0, 18.0, 38.0, 27.0, 50.0, 18.0, 46.0, 5.0, 7.315163621112927e-37, 0.0, 7.315152859140721e-37, 0.0, 7.315142097168515e-37, 0.0]
## What have you tried to solve it?
manually remove several lines from `trainining_data.lst` file, so total number of training dataset size (eg 15168) is divisible by batch size (64).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services