[GitHub] [incubator-mxnet] jmerkow opened a new issue #14421: Updating mxnet from 1.0.0, network give different ouputs

[GitBox <gi...@apache.org>]

jmerkow opened a new issue #14421: Updating mxnet from 1.0.0, network give different ouputs
URL: https://github.com/apache/incubator-mxnet/issues/14421
 
 
   I am working in a production environment, where have some networks implemented in mxnet 1.0.0. I am working updating our systems and trying to push to the latest mxnet (1.4.x as of now) but when we upgrade our networks produce different outputs.
   We are using symbols saved to json files and arg/aux_params stored in .params files. these were all produced by mxnet 1.0.0 or earlier.
   
   When using the latest mxnet (or 1.4.x) we are getting different outputs for the same inputs, with our saved models. I have been trying to use git bisect or slowly upgrading versions to figure out where this breaking change occurred but there are issues with your git history and/or some strange (compiler??) incompatibilities which prevent getting a clean checkout/build for nearly all of the intermediate versions…
   
   These are VERY different outputs, im rounding to about the 5 decimal point, or higher, so these aren’t numerical differences.
   And we are using modules, not gluons (part of the point is upgrade and move towards gluons)
   
   Does anyone have any idea what could be causing this? Or what to look into to solve it?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services