You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/06/30 10:31:13 UTC
[GitHub] [incubator-mxnet] khaotik opened a new issue #20405: mxnet cannot save parameter with name starting with slash
khaotik opened a new issue #20405:
URL: https://github.com/apache/incubator-mxnet/issues/20405
## Description
A minor issue: `mx.numpy_extension.savez` does not accept array-dict with a key starting with slash. This started to occur as soon as I upgraded to master fork. It is also inconsistent with numpy behavior.
This function is used by `gluon.Block.save_parameters` methods.
### Error Message
```text
Traceback (most recent call last):
File "test.py", line 12, in <module>
mx.numpy_extension.savez('/tmp/m_mx', **{param_name:m})
File "/media/LNXDATA/WKSP/dev/incubator-mxnet/python/mxnet/numpy_extension/utils.py", line 122, in savez
check_call(_LIB.MXNDArraySave(c_str(file), mx_uint(len(handles)), handles, keys))
File "/media/LNXDATA/WKSP/dev/incubator-mxnet/python/mxnet/base.py", line 246, in check_call
raise get_last_ffi_error()
mxnet.base.MXNetError: Traceback (most recent call last):
File "/home/khaotik/WKSP/dev/incubator-mxnet/src/serialization/cnpy.cc", line 333
MXNetError: Check failed: mz_zip_writer_add_read_buf_callback(archive, blob_name_npy.data(), npy_header_blob_read_callback, static_cast<void*>(&callback_data), size_to_add, nullptr, nullptr, 0, MZ_NO_COMPRESSION, nullptr, 0, nullptr, 0): invalid filename
```
## To Reproduce
```python
#!/usr/bin/env python
import numpy as np
import mxnet as mx
m_npy = np.random.rand(256,256).astype('float32')
m = mx.ndarray.from_numpy(m_npy)
param_name = '/op/m' # starts with a slash
np.savez('/tmp/m_np', **{param_name:m_npy}) # WORKS
mx.numpy_extension.savez('/tmp/m_mx', **{param_name:m}) # BROKEN
```
## What have you tried to solve it?
In file `python/mxnet/numpy_extension/utils.py`, substitute `NDArraySave` with `NDArrayLegacySave` provides a temp fix.
Also, the issue can be avoided by not using slash as parameter name prefix.
## Environment
ubuntu 20.04
anaconda py 3.8.3
mxnet master branch commit cb5bd4ea8b6fc9b568c13747bdb006ac047b72b5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #20405: mxnet cannot save parameter with name starting with slash
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #20405:
URL: https://github.com/apache/incubator-mxnet/issues/20405#issuecomment-871376367
npz format is based on the zip file standard. In zip files, filenames beginning with a `/` are not allowed.
> 4.4.17 file name: (Variable)
>
> 4.4.17.1 The name of the file, with optional relative path.
> The path stored MUST NOT contain a drive or
> device letter, or a leading slash. All slashes
> MUST be forward slashes '/' as opposed to
> backwards slashes '\' for compatibility with Amiga
> and UNIX file systems etc. If input came from standard
> input, there is no file name field.
https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
For reference, we're below zip implementation which, unlike Python / Numpy, enforces the filename requirements.
https://github.com/richgel999/miniz/blob/2281a42c77a08188a57609c8950b447ec62d6fb8/miniz_zip.c#L3109-L3113
Unfortunately this looks like a bug in numpy. We can consider following numpy here in being lenient about the zip file standard, but that will require additional work on the zip file implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #20405: mxnet cannot save parameter with name starting with slash
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #20405:
URL: https://github.com/apache/incubator-mxnet/issues/20405#issuecomment-871376367
npz format is based on the zip file standard. In zip files, filenames beginning with a `/` are not allowed.
> 4.4.17 file name: (Variable)
>
> 4.4.17.1 The name of the file, with optional relative path.
> The path stored MUST NOT contain a drive or
> device letter, or a leading slash. All slashes
> MUST be forward slashes '/' as opposed to
> backwards slashes '\' for compatibility with Amiga
> and UNIX file systems etc. If input came from standard
> input, there is no file name field.
https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
For reference, we're below zip implementation which, unlike Python / Numpy, enforces the filename requirements.
https://github.com/richgel999/miniz/blob/2281a42c77a08188a57609c8950b447ec62d6fb8/miniz_zip.c#L3109-L3113
Unfortunately this looks like a bug in numpy. We can consider following numpy here in writing non-standard zip files, but that's not supported by our current zip implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #20405: mxnet cannot save parameter with name starting with slash
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #20405:
URL: https://github.com/apache/incubator-mxnet/issues/20405#issuecomment-871376367
npz format is based on the zip file standard. In zip files, filenames beginning with a `/` are not allowed.
> 4.4.17 file name: (Variable)
>
> 4.4.17.1 The name of the file, with optional relative path.
> The path stored MUST NOT contain a drive or
> device letter, or a leading slash. All slashes
> MUST be forward slashes '/' as opposed to
> backwards slashes '\' for compatibility with Amiga
> and UNIX file systems etc. If input came from standard
> input, there is no file name field.
https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
For reference, we're using below zip implementation which, unlike Python / Numpy, enforces the filename requirements.
https://github.com/richgel999/miniz/blob/2281a42c77a08188a57609c8950b447ec62d6fb8/miniz_zip.c#L3109-L3113
Unfortunately this looks like a bug in numpy. We can consider following numpy here in writing non-standard zip files, but that's not supported by our current zip implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #20405: mxnet cannot save parameter with name starting with slash
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #20405:
URL: https://github.com/apache/incubator-mxnet/issues/20405#issuecomment-871376367
npz format is based on the zip file standard. In zip files, filenames beginning with a `/` are not allowed.
> 4.4.17 file name: (Variable)
>
> 4.4.17.1 The name of the file, with optional relative path.
> The path stored MUST NOT contain a drive or
> device letter, or a leading slash. All slashes
> MUST be forward slashes '/' as opposed to
> backwards slashes '\' for compatibility with Amiga
> and UNIX file systems etc. If input came from standard
> input, there is no file name field.
https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
For reference, we're below zip implementation which, unlike Python / Numpy, enforces the filename requirements.
https://github.com/richgel999/miniz/blob/2281a42c77a08188a57609c8950b447ec62d6fb8/miniz_zip.c#L3109-L3113
Unfortunately this looks like a bug in numpy. We can consider following numpy here in being lenient about the zip file standard, but that's not supported by our current zip implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org