You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/10/06 22:11:14 UTC

[GitHub] [incubator-mxnet] Zha0q1 opened a new pull request #19301: Numpy roll large tensor fix fix

Zha0q1 opened a new pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301


   This pr adds large tensor support to `np.roll`
   
   local run:
   ```
   ubuntu@ip-172-31-38-169:~/incubator-mxnet$ pytest tests/nightly/test_np_large_array.py::test_roll
   /home/ubuntu/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
     return f(*args, **kwds)
   /home/ubuntu/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
     return f(*args, **kwds)
   /home/ubuntu/anaconda3/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
     return f(*args, **kwds)
   ================================================ test session starts ================================================
   platform linux -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
   rootdir: /home/ubuntu/incubator-mxnet, inifile: pytest.ini
   plugins: remotedata-0.3.2, openfiles-0.4.0, astropy-header-0.1.2, hypothesis-5.8.3, arraydiff-0.3, doctestplus-0.5.0
   collected 1 item                                                                                                    
   
   tests/nightly/test_np_large_array.py   .                                                                        [100%]
   
   ================================================ 1 passed in 24.10s =================================================
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 closed pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 closed pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-705761259






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501953015



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       so for LTS the number of dimensions in the array is still limited to (2^31)-1, but the size of a dimension can be bigger than 2^32?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501954750



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       Yes. Number of dims has always been int. Here the original author used index_t but there is no point in doing so. Plus I think we have a hard limit for any tensors to be <= 10 dims

##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       size of individual dims and total size can both be >= 2**31 when large tensor is on

##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       Yes. Number of dims has always been int. Here the original author used index_t but there is no point in doing so. Plus I think we have a hard limit for any tensors to be <= 10 dims in many ops




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-705155515


   Jenkins CI successfully triggered : [centos-gpu, unix-cpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501953015



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       so for LTS the number of dimensions in the array is still (2^31)-1, but the size of a dimension can be bigger than 2^32?

##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       so for LTS the number of dimensions in the array is still limited to (2^31)-1, but the size of a dimension can be bigger than 2^32?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-704585099


   Jenkins CI successfully triggered : [sanity]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] samskalicky merged pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
samskalicky merged pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-704585068


   @mxnet-bot run ci [sanity]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501954750



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       Yes. Number of dims has always been int. Here the original author used index_t but there is no point in doing so. Plus I think we have a hard limit for any tensors to be <= 10 dims in many ops




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501954750



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       Yes. Number of dims has always been int. Here the original author used index_t but there is no point in doing so. Plus I think we have a hard limit for any tensors to be <= 10 dims




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] samskalicky merged pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
samskalicky merged pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501956253



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       size of individual dims and total size can both be >= 2**31 when large tensor is on




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#discussion_r501953015



##########
File path: src/operator/numpy/np_matrix_op-inl.h
##########
@@ -487,12 +487,12 @@ void NumpyRollCompute(const nnvm::NodeAttrs& attrs,
   CHECK_EQ(req.size(), 1U);
   if (inputs[0].Size() == 0U) return;
   const NumpyRollParam& param = nnvm::get<NumpyRollParam>(attrs.parsed);
-  const index_t ndim(inputs[0].shape_.ndim());
+  const int ndim(inputs[0].shape_.ndim());

Review comment:
       so for LTS the number of dimensions in the array is still (2^31)-1, but the size of a dimension can be bigger than 2^32?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-705767048


   @mseth10 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-705761259


   @samskalicky would you review and merge?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-704580286


   Hey @Zha0q1 , Thanks for submitting the PR 
   All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands: 
   - To trigger all jobs: @mxnet-bot run ci [all] 
   - To trigger specific jobs: @mxnet-bot run ci [job1, job2] 
   *** 
   **CI supported jobs**: [windows-gpu, edge, clang, unix-gpu, website, sanity, miscellaneous, unix-cpu, windows-cpu, centos-cpu, centos-gpu]
   *** 
   _Note_: 
    Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. 
   All CI tests must pass before the PR can be merged. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] Zha0q1 commented on pull request #19301: Numpy roll large tensor fix fix

Posted by GitBox <gi...@apache.org>.
Zha0q1 commented on pull request #19301:
URL: https://github.com/apache/incubator-mxnet/pull/19301#issuecomment-705155465


   @mxnet-bot run ci [centos-gpu, unix-cpu]


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org