You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/04/01 10:54:30 UTC

[GitHub] [incubator-mxnet] konioyxgq opened a new issue #17953: Models saved at different training stages with different forward speeds

konioyxgq opened a new issue #17953: Models saved at different training stages with different forward speeds
URL: https://github.com/apache/incubator-mxnet/issues/17953
 
 
   C++ interface under Version 1.3.1 . During training, the first-time saved model, the forward time is 0.2s. But the model saved when the training is complete, the forward time is 0.8s.
   What causes this problem? How to fix it? anyone can give some advises? Thanks!
   
   envs: cpu/c++/mxnet1.3.1

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] konioyxgq commented on issue #17953: Models saved at different training stages with different forward speeds

Posted by GitBox <gi...@apache.org>.
konioyxgq commented on issue #17953: Models saved at different training stages with different forward speeds
URL: https://github.com/apache/incubator-mxnet/issues/17953#issuecomment-609522162
 
 
   Thank you again for your patient response. Thank you very much!!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] konioyxgq closed issue #17953: Models saved at different training stages with different forward speeds

Posted by GitBox <gi...@apache.org>.
konioyxgq closed issue #17953: Models saved at different training stages with different forward speeds
URL: https://github.com/apache/incubator-mxnet/issues/17953
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] konioyxgq commented on issue #17953: Models saved at different training stages with different forward speeds

Posted by GitBox <gi...@apache.org>.
konioyxgq commented on issue #17953: Models saved at different training stages with different forward speeds
URL: https://github.com/apache/incubator-mxnet/issues/17953#issuecomment-609382181
 
 
   Thank you very much for your reply.
   This situation occurs. Is it because of what went wrong with my training? Or is there a problem with my model? Do you have any suggestions?
   Thank you very much！
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] RuRo commented on issue #17953: Models saved at different training stages with different forward speeds

Posted by GitBox <gi...@apache.org>.
RuRo commented on issue #17953: Models saved at different training stages with different forward speeds
URL: https://github.com/apache/incubator-mxnet/issues/17953#issuecomment-609412044
 
 
   I don't think, this situation necessarily means, that there is something wrong with your training/model. It just means that some weights are slowly converging to 0s during training.
   
   If you are worried about your models correctness, you can manually check, which parameters are close to 0 after training. But unless it's some obvious problem, I don't think, that there's something wrong with parameters converging to 0s.
   
   In fact, some regularization techniques explicitly aim for such convergence because if you have a whole row or column of 0s in your weights, you can remove that feature from your network without changing the output, while also saving on computation complexity.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-mxnet] RuRo commented on issue #17953: Models saved at different training stages with different forward speeds

Posted by GitBox <gi...@apache.org>.
RuRo commented on issue #17953: Models saved at different training stages with different forward speeds
URL: https://github.com/apache/incubator-mxnet/issues/17953#issuecomment-608949381
 
 
   Hi. I've previously had a similar problem, which turned out to be due to [denormal numbers](https://en.wikipedia.org/wiki/Denormal_number) in the model weights. This issue only affects inference times on the CPU, since on some modern CPUs floating-point computations are significantly slower for denormal numbers.
   
   In my experience, denormal numbers often appear in model weights, when you have vanishing gradients and/or recurrent neural connections. The fix for me was to simply manually set all the denormalized weights to 0. This shouldn't significantly change the outputs of your model since the denormalized numbers are so small in the first place.
   
   ```python
   import numpy as np
   import mxnet as mx
   
   model = com.load_hybrid(model_path)
   for p, v in model.collect_params().items():
       vd = v.data().asnumpy()
       vd[np.abs(vd) < 1e-37] = 0.0
       v.set_data(mx.nd.array(vd))
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services