[GitHub] zhreshold opened a new issue #7368: Caffe converter test fails, causing CI to halt for all PRs

[gi...@git.apache.org]

zhreshold opened a new issue #7368: Caffe converter test fails, causing CI to halt for all PRs
URL: https://github.com/apache/incubator-mxnet/issues/7368
 
 
   ## Steps to reproduce
   or if you are running standard examples, please provide the commands you have run that lead to the error.
   
   1. python tools/caffe_converter/test_converter.py
   2.
   3.
   
   ## What have you tried to solve it?
   Tried to debug the metric_update process, figured out for vgg-16 and resnet, the network prediction output is wrong, we have the last conv output mixed to the final prediction. Thus causing the mismatched shape with label.
   ```
   ('!!!labels', 1, (32L,))
   ('!!!preds', 2, (32L, 1000L), (32L, 512L, 7L, 7L))
   ```
   For googlenet, the output is correct:
   ```
   ('!!!labels', 1, (32L,))
   ('!!!preds', 2, (32L, 1000L))
   ```
   @joey2014 
   https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/master/152/pipeline/
   
   ```
   INFO:root:Saved checkpoint to "./model/vgg-16-0000.params"
   data/val-5k-256.rec
   [18:21:16] src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: data/val-5k-256.rec, use 4 threads for decoding..
   ('!!!labels', 1, (32L,))
   ('!!!preds', 2, (32L, 1000L), (32L, 512L, 7L, 7L))
   Traceback (most recent call last):
     File "test_converter.py", line 89, in <module>
       main()
     File "test_converter.py", line 86, in main
       test_imagenet_model_performance(m, val, gpus, batch_size)
     File "test_converter.py", line 38, in test_imagenet_model_performance
       **mean_args)
     File "/home/ubuntu/debug/incubator-mxnet/tools/caffe_converter/../../example/image-classification/score.py", line 60, in score
       mod.update_metric(m, batch.label)
     File "/home/ubuntu/verify/mxnet/python/mxnet/module/module.py", line 718, in update_metric
       self._exec_group.update_metric(eval_metric, labels)
     File "/home/ubuntu/verify/mxnet/python/mxnet/module/executor_group.py", line 565, in update_metric
       eval_metric.update_dict(labels_, preds)
     File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 91, in update_dict
       self.update(label, pred)
     File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 373, in update
       check_label_shapes(labels, preds)
     File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 24, in check_label_shapes
       "predictions {}".format(label_shape, pred_shape))
   ValueError: Shape of labels 1 does not match shape of predictions 2
   ```
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services