You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/01/22 22:36:55 UTC
[GitHub] [incubator-mxnet] access2rohit opened a new pull request #17410:
skipping flaky tests randint, log_softmax. Added python gc to teardown.
access2rohit opened a new pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410
## Description ##
skipping flaky tests randint,log_softmax. Added python gc to teardown. Skipped topk due to large memory usage
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [x] Changes are complete (i.e. I finished coding on this PR)
- [x] All changes have test coverage:
- Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
- Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
- Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
- [x] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
## Testing ##
```
MXNET_TEST_COUNT=1 nosetests --logging-level=DEBUG --verbose -s tests/nightly/test_large_array.py:test_tensor
/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
test_large_array.test_tensor ... [DEBUG] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=215611752 to reproduce.
SKIP: Randint flaky, tracked at https://github.com/apache/incubator-mxnet/issues/16172
----------------------------------------------------------------------
Ran 1 test in 125.584s
OK (SKIP=1)
MXNET_TEST_COUNT=1 nosetests --logging-level=DEBUG --verbose -s tests/nightly/test_large_array.py:test_nn
/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
test_large_array.test_nn ... [21:42:14] src/executor/graph_executor.cc:2062: Subgraph backend MKLDNN is activated.
[21:42:18] src/executor/graph_executor.cc:2062: Subgraph backend MKLDNN is activated.
SKIP: log_softmax flaky, tracked at https://github.com/apache/incubator-mxnet/issues/17397
----------------------------------------------------------------------
Ran 1 test in 444.677s
OK (SKIP=1)
MXNET_TEST_COUNT=1 nosetests --logging-level=DEBUG --verbose -s tests/nightly/test_large_array.py:test_basic
/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
test_large_array.test_basic ... SKIP: Topk takes lot of memory!, disabling it for now!
----------------------------------------------------------------------
Ran 1 test in 1254.173s
OK (SKIP=1)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] access2rohit commented on a change in pull
request #17410: skipping flaky tests randint,
log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369852113
##########
File path: tests/python/unittest/common.py
##########
@@ -308,6 +309,7 @@ def teardown():
It waits for all operations in one file to finish before carrying on the next.
"""
mx.nd.waitall()
+ gc.collect()
Review comment:
Actually not sure. I didn't know if there was a way to identify whether it was helpful or not. In a nutshell I still ran out of memory for 480GB machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] access2rohit commented on a change in pull
request #17410: skipping flaky tests randint,
log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369852113
##########
File path: tests/python/unittest/common.py
##########
@@ -308,6 +309,7 @@ def teardown():
It waits for all operations in one file to finish before carrying on the next.
"""
mx.nd.waitall()
+ gc.collect()
Review comment:
Actually not sure. I didn't know if there was a way to identify whether it was helpful or not. In a nutshell I still ran out of memory on a 480GB memory machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] access2rohit commented on issue #17410: skipping
flaky tests randint, log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
access2rohit commented on issue #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#issuecomment-577418954
@mxnet-label-bot add [pr-awaiting-review]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request
#17410: skipping flaky tests randint,
log_softmax for large tensor. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #17410: skipping flaky tests randint,log_softmax for large tensor. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369862370
##########
File path: tests/python/unittest/common.py
##########
@@ -308,6 +309,7 @@ def teardown():
It waits for all operations in one file to finish before carrying on the next.
"""
mx.nd.waitall()
+ gc.collect()
Review comment:
Have played with teardown multiple times before.
Learnt only 1 thing : MXNet Memory Management would be a great project :p
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request
#17410: skipping flaky tests randint,
log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
apeforest commented on a change in pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369847719
##########
File path: tests/python/unittest/common.py
##########
@@ -31,6 +31,7 @@
from contextlib import contextmanager
from nose.tools import make_decorator, assert_raises
import tempfile
+import gc
Review comment:
nit: we usually import generic modules before all the custom modules. So this is better to go in the second line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] access2rohit commented on a change in pull
request #17410: skipping flaky tests randint,
log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
access2rohit commented on a change in pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369852132
##########
File path: tests/python/unittest/common.py
##########
@@ -31,6 +31,7 @@
from contextlib import contextmanager
from nose.tools import make_decorator, assert_raises
import tempfile
+import gc
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] access2rohit commented on issue #17410: skipping
flaky tests randint, log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
access2rohit commented on issue #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#issuecomment-577419193
@apeforest @ChaiBapchya PR ready for review. Had to skip topk() in order to execute all tests in one go.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request
#17410: skipping flaky tests randint,
log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
apeforest commented on a change in pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369847110
##########
File path: tests/nightly/test_large_array.py
##########
@@ -1089,6 +1092,7 @@ def check_sort():
s = nd.sort(b, is_ascend=False)
assert np.sum(s[0].asnumpy() == 0).all()
+ @unittest.skip("Topk takes lot of memory!, disabling it for now!")
Review comment:
Can you create an issue to track this and add in the clause here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest commented on a change in pull request
#17410: skipping flaky tests randint,
log_softmax. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
apeforest commented on a change in pull request #17410: skipping flaky tests randint,log_softmax. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410#discussion_r369847269
##########
File path: tests/python/unittest/common.py
##########
@@ -308,6 +309,7 @@ def teardown():
It waits for all operations in one file to finish before carrying on the next.
"""
mx.nd.waitall()
+ gc.collect()
Review comment:
Just curious. Did this actually help?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] apeforest merged pull request #17410: skipping
flaky tests randint,
log_softmax for large tensor. Added python gc to teardown.
Posted by GitBox <gi...@apache.org>.
apeforest merged pull request #17410: skipping flaky tests randint,log_softmax for large tensor. Added python gc to teardown.
URL: https://github.com/apache/incubator-mxnet/pull/17410
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services