You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/05/19 20:20:33 UTC

[GitHub] [incubator-mxnet] szha opened a new pull request #18368: [WIP] enable large tensor in np

szha opened a new pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368


   ## Description ##
   (Brief description on what this PR is about)
   
   ## Checklist ##
   ### Essentials ###
   Please feel free to remove inapplicable items for your PR.
   - [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments are documented. 
   - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
   - Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
   - [ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be made.
   - Interesting edge cases to note here
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456628889



##########
File path: tests/python/unittest/test_np_large_array.py
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import tempfile
+import math
+import numpy as np
+import mxnet as mx
+
+curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+sys.path.append(os.path.join(curr_path, '../python/unittest/'))
+
+from mxnet.test_utils import rand_ndarray, assert_almost_equal, rand_coord_2d, default_context, check_symbolic_forward, create_2d_tensor, use_np
+from mxnet import gluon, nd
+from common import with_seed
+import pytest
+
+
+# dimension constants
+SMALL_X = 100
+MEDIUM_X = 10000
+LARGE_X = 100000000
+VLARGE_X = 4300000000
+SMALL_Y = 50
+
+
+@use_np
+def test_gluon_embedding():
+    m = gluon.nn.Embedding(SMALL_Y, MEDIUM_X)
+    m.initialize()
+    a = mx.np.zeros((MEDIUM_X, SMALL_Y))
+    b = m(a)
+    assert b.shape == (MEDIUM_X, SMALL_Y, MEDIUM_X)
+    assert b.asnumpy().size == LARGE_X * SMALL_Y
+
+@use_np
+def test_fully_connected():
+    a = mx.np.ones(shape=(LARGE_X, SMALL_Y))
+    b = mx.np.ones(shape=(SMALL_Y, SMALL_Y))
+    c = mx.np.ones(shape=(b.shape[0],))
+
+    # w/o bias
+    res = mx.npx.FullyConnected(a, b, num_hidden=b.shape[0], no_bias=True)
+    assert np.sum(res[-1].asnumpy() == a.shape[1]) == b.shape[0]
+
+    # w/ bias
+    res = mx.npx.FullyConnected(a, b, c, num_hidden=b.shape[0], no_bias=False)
+    assert np.sum(res[-1].asnumpy() == a.shape[1] + 1) == b.shape[0]
+
+@use_np
+def test_dense():
+    data = mx.np.ones(shape=(50*1000*1000, 100))
+    linear = gluon.nn.Dense(100)
+    linear.initialize()
+    res = linear(data)
+    assert res.shape == (50000000, 100)

Review comment:
       yes will do




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456611407



##########
File path: python/mxnet/numpy/multiarray.py
##########
@@ -40,8 +40,9 @@
                       get_oshape_of_gather_nd_op
 from ..ndarray._internal import _set_np_ndarray_class
 from . import _op as _mx_np_op
-from ..base import check_call, _LIB, NDArrayHandle, c_array
+from ..base import check_call, _LIB, NDArrayHandle, c_array, mx_int, mx_int64

Review comment:
       This is actually correct no need to change it. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-659852424


   @sandeep-krishnamurthy the CI is currently stuck on build issue that @TaoLv pointed out. @access2rohit 's PR on the build fix is needed to address the issue but it seems that there has been no progress in #18645. What's the plan?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456618261



##########
File path: python/mxnet/numpy/multiarray.py
##########
@@ -92,6 +93,16 @@
 _NDARRAY_NO_ZERO_DIM_BOOL_ARRAY = -1
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_FALSE = 0
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_TRUE = 1
+_SIGNED_INT32_UPPER_LIMIT = (2**31 - 1)

Review comment:
       since you are importing ndarray.ndarray here: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/numpy/multiarray.py#L51  
   `_SIGNED_INT32_UPPER_LIMIT` is already defined in ndarray.py file: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/ndarray/ndarray.py#L60 
   and can be imported like this: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/symbol/symbol.py#L40
   So this won't be necessary. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-660292353


   @sandeep-krishnamurthy I'm not asking about MKL as feature request. I'm saying that the current build is broken and it needs fix.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on pull request #18368: [NumPy] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-661169139


   @access2rohit thanks for the review. no codecov isn't enforced yet.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-660294002


   > @sandeep-krishnamurthy I'm not asking about MKL as feature request. I'm saying that the current build is broken and it needs fix.
   
   Basically I will disable large tensor build with MKL in CI when enabling Large Tensor by default and that should fix it(Since we are currently focusing on making Large Tensor work with openBLAS). For now you can remove your change to make USE_INT64_TENSOR_SIZE=ON as default from CMakeLists.txt and your PR should pass CI. Sounds good ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456629049



##########
File path: python/mxnet/numpy/multiarray.py
##########
@@ -92,6 +93,16 @@
 _NDARRAY_NO_ZERO_DIM_BOOL_ARRAY = -1
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_FALSE = 0
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_TRUE = 1
+_SIGNED_INT32_UPPER_LIMIT = (2**31 - 1)
+
+# Caching whether MXNet was built with INT64 support or not
+_INT64_TENSOR_SIZE_ENABLED = None
+
+def _int64_enabled():
+    global _INT64_TENSOR_SIZE_ENABLED
+    if _INT64_TENSOR_SIZE_ENABLED is None:
+        _INT64_TENSOR_SIZE_ENABLED = Features().is_enabled('INT64_TENSOR_SIZE')
+    return _INT64_TENSOR_SIZE_ENABLED

Review comment:
       we are deleting ndarray later so the copy is intentional.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha merged pull request #18368: [NumPy] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha merged pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] TaoLv commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
TaoLv commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-653774331


   @szha, do you mean the errors below? As I mentioned in https://github.com/apache/incubator-mxnet/pull/18645#issuecomment-652198544, we need also enable `MKL_USE_ILP64` in cmake line when LTS is enabled. More information, see https://software.intel.com/content/www/us/en/develop/documentation/mkl-macos-developer-guide/top/linking-your-application-with-the-intel-math-kernel-library/linking-in-detail/linking-with-interface-libraries/using-the-ilp64-interface-vs-lp64-interface.html. I'm afraid that similar problem may also exist in mshadow code: https://github.com/apache/incubator-mxnet/blob/master/3rdparty/mshadow/mshadow/dot_engine-inl.h#L317.
   
   ```
   [2020-07-03T21:00:50.972Z] /work/mxnet/src/operator/contrib/transformer.cc:143:31: error: narrowing conversion of 'm' from 'mxnet::index_t {aka long int}' to 'int' inside { } [-Werror=narrowing]
   [2020-07-03T21:00:50.972Z]    MKL_INT p_m[GROUP_SIZE] = {m};
   [2020-07-03T21:00:50.972Z]                                ^
   [2020-07-03T21:00:50.972Z] /work/mxnet/src/operator/contrib/transformer.cc:144:31: error: narrowing conversion of 'n' from 'mxnet::index_t {aka long int}' to 'int' inside { } [-Werror=narrowing]
   [2020-07-03T21:00:50.972Z]    MKL_INT p_n[GROUP_SIZE] = {n};
   [2020-07-03T21:00:50.972Z]                                ^
   [2020-07-03T21:00:50.972Z] /work/mxnet/src/operator/contrib/transformer.cc:145:31: error: narrowing conversion of 'k' from 'mxnet::index_t {aka long int}' to 'int' inside { } [-Werror=narrowing]
   [2020-07-03T21:00:50.972Z]    MKL_INT p_k[GROUP_SIZE] = {k};
   [2020-07-03T21:00:50.972Z]                                ^
   [2020-07-03T21:00:50.972Z] /work/mxnet/src/operator/contrib/transformer.cc:146:35: error: narrowing conversion of 'lda' from 'mxnet::index_t {aka long int}' to 'int' inside { } [-Werror=narrowing]
   [2020-07-03T21:00:50.972Z]    MKL_INT p_lda[GROUP_SIZE] = {lda};
   [2020-07-03T21:00:50.972Z]                                    ^
   [2020-07-03T21:00:50.972Z] /work/mxnet/src/operator/contrib/transformer.cc:147:35: error: narrowing conversion of 'ldb' from 'mxnet::index_t {aka long int}' to 'int' inside { } [-Werror=narrowing]
   [2020-07-03T21:00:50.972Z]    MKL_INT p_ldb[GROUP_SIZE] = {ldb};
   [2020-07-03T21:00:50.972Z]                                    ^
   [2020-07-03T21:00:50.972Z] /work/mxnet/src/operator/contrib/transformer.cc:148:35: error: narrowing conversion of 'ldc' from 'mxnet::index_t {aka long int}' to 'int' inside { } [-Werror=narrowing]
   [2020-07-03T21:00:50.972Z]    MKL_INT p_ldc[GROUP_SIZE] = {ldc};
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456629856



##########
File path: python/mxnet/numpy/multiarray.py
##########
@@ -40,8 +40,9 @@
                       get_oshape_of_gather_nd_op
 from ..ndarray._internal import _set_np_ndarray_class
 from . import _op as _mx_np_op
-from ..base import check_call, _LIB, NDArrayHandle, c_array
+from ..base import check_call, _LIB, NDArrayHandle, c_array, mx_int, mx_int64

Review comment:
       mx_int/mx_int64 are used below.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-660292641


   @access2rohit will do. We still need to fix the CI for enabling large tensor first.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit edited a comment on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit edited a comment on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-660291731


   @szha 
   Can we move the file: `tests/python/unittest/test_np_large_array.py` to nightly folder instead? Keeping them in unittest will significantly slowdown the CI and may result in timeouts as we are allocating tensors with over 4.3 Billion elements. Even if CI doesn't timeout right now once all numpy ops are added it will definitely timeout. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-653812750


   @TaoLv thanks. I will wait for that PR to be resolved.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-659884801


   As discussed in this issue https://github.com/apache/incubator-mxnet/issues/17331#issuecomment-656725373 as a first step we are focussed on OpenBLAS updates only as it is the primary BLAS engine we ship with PyPi. MKL updates is next.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-631060274


   Hey @szha , Thanks for submitting the PR 
   All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands: 
   - To trigger all jobs: @mxnet-bot run ci [all] 
   - To trigger specific jobs: @mxnet-bot run ci [job1, job2] 
   *** 
   **CI supported jobs**: [centos-gpu, windows-gpu, miscellaneous, clang, unix-cpu, website, unix-gpu, centos-cpu, sanity, edge, windows-cpu]
   *** 
   _Note_: 
    Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin. 
   All CI tests must pass before the PR can be merged. 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-660292618


   Overall functionality seems fine. Can you run all the tests in the file `tests/python/unittest/test_np_large_array.py` using pytest and paste the output in the comments section so we can be sure that nightly CD pipeline doesn't break.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456629152



##########
File path: python/mxnet/numpy/multiarray.py
##########
@@ -92,6 +93,16 @@
 _NDARRAY_NO_ZERO_DIM_BOOL_ARRAY = -1
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_FALSE = 0
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_TRUE = 1
+_SIGNED_INT32_UPPER_LIMIT = (2**31 - 1)

Review comment:
       we are deleting ndarray later so the copy is intentional.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456624909



##########
File path: tests/python/unittest/test_np_large_array.py
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import tempfile
+import math
+import numpy as np
+import mxnet as mx
+
+curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+sys.path.append(os.path.join(curr_path, '../python/unittest/'))
+
+from mxnet.test_utils import rand_ndarray, assert_almost_equal, rand_coord_2d, default_context, check_symbolic_forward, create_2d_tensor, use_np
+from mxnet import gluon, nd
+from common import with_seed
+import pytest
+
+
+# dimension constants
+SMALL_X = 100
+MEDIUM_X = 10000
+LARGE_X = 100000000
+VLARGE_X = 4300000000
+SMALL_Y = 50
+
+
+@use_np
+def test_gluon_embedding():
+    m = gluon.nn.Embedding(SMALL_Y, MEDIUM_X)
+    m.initialize()
+    a = mx.np.zeros((MEDIUM_X, SMALL_Y))
+    b = m(a)
+    assert b.shape == (MEDIUM_X, SMALL_Y, MEDIUM_X)
+    assert b.asnumpy().size == LARGE_X * SMALL_Y
+
+@use_np
+def test_fully_connected():
+    a = mx.np.ones(shape=(LARGE_X, SMALL_Y))
+    b = mx.np.ones(shape=(SMALL_Y, SMALL_Y))
+    c = mx.np.ones(shape=(b.shape[0],))
+
+    # w/o bias
+    res = mx.npx.FullyConnected(a, b, num_hidden=b.shape[0], no_bias=True)
+    assert np.sum(res[-1].asnumpy() == a.shape[1]) == b.shape[0]
+
+    # w/ bias
+    res = mx.npx.FullyConnected(a, b, c, num_hidden=b.shape[0], no_bias=False)
+    assert np.sum(res[-1].asnumpy() == a.shape[1] + 1) == b.shape[0]
+
+@use_np
+def test_dense():
+    data = mx.np.ones(shape=(50*1000*1000, 100))
+    linear = gluon.nn.Dense(100)
+    linear.initialize()
+    res = linear(data)
+    assert res.shape == (50000000, 100)

Review comment:
       Can we use dimension constants instead of hard coded values ? I do see the its not correctly written in test_large_array.py(https://github.com/apache/incubator-mxnet/blob/master/tests/nightly/test_large_array.py#L68-L73) I will fix it there




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-659620714


   @access2rohit - Please help review. Thanks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-660291731


   @szha 
   Can we move the file: tests/python/unittest/test_np_large_array.py to nightly folder instead? Keeping them in unittest will significantly slowdown the CI and may result in timeouts as we are allocating tensors with over 4.3 Billion elements. Even if CI doesn't timeout right now once all numpy ops are added it will definitely timeout. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] szha commented on pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
szha commented on pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#issuecomment-653678603


   @TaoLv @PatricZhao large tensor build for MKL builds seems to be failing. See the CI checks.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-mxnet] access2rohit commented on a change in pull request #18368: [WIP] enable large tensor in np

Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #18368:
URL: https://github.com/apache/incubator-mxnet/pull/18368#discussion_r456620176



##########
File path: python/mxnet/numpy/multiarray.py
##########
@@ -92,6 +93,16 @@
 _NDARRAY_NO_ZERO_DIM_BOOL_ARRAY = -1
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_FALSE = 0
 _NDARRAY_ZERO_DIM_BOOL_ARRAY_TRUE = 1
+_SIGNED_INT32_UPPER_LIMIT = (2**31 - 1)
+
+# Caching whether MXNet was built with INT64 support or not
+_INT64_TENSOR_SIZE_ENABLED = None
+
+def _int64_enabled():
+    global _INT64_TENSOR_SIZE_ENABLED
+    if _INT64_TENSOR_SIZE_ENABLED is None:
+        _INT64_TENSOR_SIZE_ENABLED = Features().is_enabled('INT64_TENSOR_SIZE')
+    return _INT64_TENSOR_SIZE_ENABLED

Review comment:
       Already define here: https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/ndarray/ndarray.py#L122-L128
   
   We can just import it since are already importing functions from that file 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org