[GitHub] FCInter opened a new issue #13783: Predict result is bad when setting for_training=False

[GitBox <gi...@apache.org>]

FCInter opened a new issue #13783: Predict result is bad when setting for_training=False
URL: https://github.com/apache/incubator-mxnet/issues/13783
 
 
   ## Description
   I trained a model and used it to perform prediction. While building the predictor, if I set the argument for_training=False, the prediction result is bad, as bad as predicted using a randomly initialized model.
   
   ## Environment info (Required)
   
   ```
   What to do:
   ----------Python Info----------
   ('Version      :', '2.7.12')
   ('Compiler     :', 'GCC 5.4.0 20160609')
   ('Build        :', ('default', 'Dec  4 2017 14:50:18'))
   ('Arch         :', ('64bit', ''))
   ------------Pip Info-----------
   ('Version      :', '18.1')
   ('Directory    :', '/path/to/mx_env/local/lib/python2.7/site-packages/pip')
   ----------MXNet Info-----------
   ('Version      :', '1.3.0')
   ('Directory    :', '/path/to/mx_env/local/lib/python2.7/site-packages/mxnet')
   ('Commit Hash   :', 'b3be92f4a48bce62a5a8424271871c2f81c8f7f1')
   ----------System Info----------
   ('Platform     :', 'Linux-4.4.0-87-generic-x86_64-with-Ubuntu-16.04-xenial')
   ('system       :', 'Linux')
   ('node         :', 'B22-C09-G5500-01-GPU')
   ('release      :', '4.4.0-87-generic')
   ('version      :', '#110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017')
   ----------Hardware Info----------
   ('machine      :', 'x86_64')
   ('processor    :', 'x86_64')
   Architecture:          x86_64
   CPU op-mode(s):        32-bit, 64-bit
   Byte Order:            Little Endian
   CPU(s):                88
   On-line CPU(s) list:   0-87
   Thread(s) per core:    2
   Core(s) per socket:    22
   Socket(s):             2
   NUMA node(s):          2
   Vendor ID:             GenuineIntel
   CPU family:            6
   Model:                 79
   Model name:            Intel(R) Xeon(R) CPU E5-2699A v4 @ 2.40GHz
   Stepping:              1
   CPU MHz:               2400.093
   CPU max MHz:           3600.0000
   CPU min MHz:           1200.0000
   BogoMIPS:              4801.21
   Virtualization:        VT-x
   Hypervisor vendor:     vertical
   Virtualization type:   full
   L1d cache:             32K
   L1i cache:             32K
   L2 cache:              256K
   L3 cache:              56320K
   NUMA node0 CPU(s):     0-21,44-65
   NUMA node1 CPU(s):     22-43,66-87
   
   
   Package used (Python/R/Scala/Julia):
   Python
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio):
   gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
   
   Build config:
   I use pip install.
   
   ## Error Message:
   No error message. The problem happens to the prediction results. Please go to the Minimum reproducible example.
   
   ## Minimum reproducible example
   This is how I built the predictor:
   
   ```
   class Predictor(object):
       def __init__(self, symbol, data_names, label_names,
                    context=mx.cpu(), max_data_shapes=None,
                    provide_data=None, provide_label=None,
                    arg_params=None, aux_params=None):
           self._mod = MutableModule(symbol, data_names, label_names,
                                     context=context, max_data_shapes=max_data_shapes)
           self._mod.bind(provide_data, provide_label, for_training=True)
           self._mod.init_params(arg_params=arg_params, aux_params=aux_params)
   ```
   
   The problem happens to second last line, i.e. `self._mod.bind(provide_data, provide_label, for_training=True)`. If I set the `for_training=True`, the prediction results look good. But if I set `for_training=False`, the prediction results look quite bad. Not a single object was detected on all hundreds of test images. It looks like if I set `for_training=False`, the model parameters are randomly initialized, instead of initialized from the saved `.params` file.
   
   ## What have you tried to solve it?
   
   I tried to get the parameters after `init_params`, i.e. after `init_params`, I add `arg_params, aux_params = self._mod.get_params()`, and then use `self._mod.save_params(tar_filename)`, save the params to files. I saved both the params of `for_training=True` and `for_training=False`. Strangely, the two save params are exactly equal.
   
   What's wrong with my code?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services