You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/04/27 17:20:16 UTC
[GitHub] [incubator-mxnet] brightening-eyes opened a new issue #20225: ability of saving parameters to memory
brightening-eyes opened a new issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225
## Description
the save_parameters() method of gluon.nn.HybridBlock only saves model parameters to file. it should be possible to save the model to a file-like object. this lets us combine many models, save them to a dict object, pickle that dict object and load from that (think of saving encoder and decoder into just one file than compress it).
of course the possibilities are endless, and this was just an example.
## References
pytorch has an api torch.save and torch.load as far as I can remember.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #20225: ability of saving parameters to memory
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225#issuecomment-829527892
There may be issues during unpickle on other systems, for example when the systems differ in endianness. I suggest to convert each param to numpy first and then pickle. You can use pickle out-of-band data support for efficiency: https://www.python.org/dev/peps/pep-0574/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] brightening-eyes commented on issue #20225: ability of saving parameters to memory
Posted by GitBox <gi...@apache.org>.
brightening-eyes commented on issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225#issuecomment-831363282
@leezu I want to pickle with x86_64, and unpickle using that architecture. so, that should not be an issue. but, my main concern was the ability of saving more than 1 network + additional data like the parameters of the optimizer into 1 file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #20225: ability of saving parameters to memory
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225#issuecomment-828042967
You can collect the parameters and convert them to numpy.
Also note that as of MXNet v2, the saves to the numpy npz file format (see https://numpy.org/doc/stable/reference/generated/numpy.savez.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #20225: ability of saving parameters to memory
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225#issuecomment-829527892
If you pickle the mxnet parameters, there may be issues during unpickle on other systems, for example when the systems differ in endianness. Thus it's safer to convert each param to numpy first and then pickle. You can use pickle out-of-band data support for efficiency: https://www.python.org/dev/peps/pep-0574/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #20225: ability of saving parameters to memory
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225#issuecomment-829527892
If you pickle the mxnet parameters, there may be issues during unpickle on other systems, for example when the systems differ in endianness. I suggest to convert each param to numpy first and then pickle. You can use pickle out-of-band data support for efficiency: https://www.python.org/dev/peps/pep-0574/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] brightening-eyes commented on issue #20225: ability of saving parameters to memory
Posted by GitBox <gi...@apache.org>.
brightening-eyes commented on issue #20225:
URL: https://github.com/apache/incubator-mxnet/issues/20225#issuecomment-829252894
@leezu is it possible to pickle the collected parameters without any data loss?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org