You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/04/11 01:09:40 UTC
[GitHub] [incubator-mxnet] szha opened a new pull request #18025: [DEV]
switch nose with pytest
szha opened a new pull request #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025
## Description ##
switch nose with pytest
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage:
- Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
- Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
- Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
- [x] Code is well-documented
- [x] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
### Changes ###
- [x] switch nose with pytest
## Comments ##
- If this change is a backward incompatible change, why must this change be made.
- Interesting edge cases to note here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on a change in pull request
#18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r409223358
##########
File path: .github/workflows/os_x_staticbuild.yml
##########
@@ -10,16 +10,16 @@ jobs:
uses: actions/checkout@v2
- name: Install Dependencies
run: |
- brew install nasm automake ninja libtool cmake pkgconfig protobuf
+ brew install nasm automake ninja libtool cmake pkgconfig protobuf hdf5 zlib
+ python3 -m pip install --user -r ci/docker/install/requirements
Review comment:
I suggest not to use `ci/docker/install/requirements` here but list the dependencies explicitly. Most of the things in `ci/docker/install/requirements` are not needed for this test and having a single requirements file used for a that is used from an opaque number of locations for an variety of purposes makes it very hard to maintain that file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] mxnet-bot commented on issue #18025: [DEV] switch
nose with pytest
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on issue #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-612287468
Hey @szha , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [centos-gpu, edge, windows-cpu, website, centos-cpu, unix-cpu, unix-gpu, windows-gpu, miscellaneous, sanity, clang]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] szha commented on a change in pull request
#18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r409275673
##########
File path: .github/workflows/os_x_staticbuild.yml
##########
@@ -10,16 +10,16 @@ jobs:
uses: actions/checkout@v2
- name: Install Dependencies
run: |
- brew install nasm automake ninja libtool cmake pkgconfig protobuf
+ brew install nasm automake ninja libtool cmake pkgconfig protobuf hdf5 zlib
Review comment:
this was for installing h5py and pillow, which triggers compile in source distribution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] mxnet-bot commented on issue #18025: [DEV] switch
nose with pytest
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on issue #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616054747
Jenkins CI successfully triggered : [windows-cpu, unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch
nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616054717
@mxnet-bot run ci [unix-cpu, windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch
nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-612309366
How about the random seed handling and reproducibility? I think you'll need to adapt https://github.com/dmlc/gluon-nlp/blob/v0.9.x/conftest.py#L41
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] szha commented on issue #18025: [DEV] switch nose
with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-612348483
@leezu thanks. Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on a change in pull request
#18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r409224226
##########
File path: .github/workflows/os_x_staticbuild.yml
##########
@@ -10,16 +10,16 @@ jobs:
uses: actions/checkout@v2
- name: Install Dependencies
run: |
- brew install nasm automake ninja libtool cmake pkgconfig protobuf
+ brew install nasm automake ninja libtool cmake pkgconfig protobuf hdf5 zlib
Review comment:
Why are the dependencies on hdf5 zlib introduced when switching to pytest?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r418271451
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -994,26 +991,24 @@ cd_unittest_ubuntu() {
local mxnet_variant=${1:?"This function requires a mxnet variant as the first argument"}
- local nose_cmd="nosetests-3.4"
-
- $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/unittest
- $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/quantization
+ pytest --durations=50 --verbose tests/python/unittest
+ pytest --durations=50 --verbose tests/python/quantization
# https://github.com/apache/incubator-mxnet/issues/11801
# if [[ ${mxnet_variant} = "cpu" ]] || [[ ${mxnet_variant} = "mkl" ]]; then
# integrationtest_ubuntu_cpu_dist_kvstore
# fi
if [[ ${mxnet_variant} = cu* ]]; then
- $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/gpu
+ pytest --durations=50 --verbose tests/python/gpu
Review comment:
why are we showing top 50 slowest duration only?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411678829
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
Review comment:
Ah, it seems like the test seed value is actually printed as I requested here: https://github.com/apache/incubator-mxnet/pull/18025/files#diff-c9b030b0828c1291b56fc10aba6baab6R204
Is that correct?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411675629
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
Review comment:
Would it be possible to simplify this process? It seems like multiple iterations would be required to get the correct seed.
Instead, would it be possible to print the used seeds after a test failed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617278498
and github was down briefly 🤦♂️
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request
#18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18025: [DEV] switch nose with pytest
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r409223358
##########
File path: .github/workflows/os_x_staticbuild.yml
##########
@@ -10,16 +10,16 @@ jobs:
uses: actions/checkout@v2
- name: Install Dependencies
run: |
- brew install nasm automake ninja libtool cmake pkgconfig protobuf
+ brew install nasm automake ninja libtool cmake pkgconfig protobuf hdf5 zlib
+ python3 -m pip install --user -r ci/docker/install/requirements
Review comment:
I suggest not to use `ci/docker/install/requirements` here but list the dependencies explicitly. Most of the things in `ci/docker/install/requirements` are not needed for this test and having a single requirements file that is used from an opaque number of locations for an variety of purposes makes it very hard to maintain that file.
Specifically, here we can just replace `python3 -m pip install --user nose nose-timer nose-exclude numpy scipy` with `python3 -m pip install --user pytest numpy scipy` a few lines below.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617295296
> An option would of course be to cancel the active run, but that's a separate question I guess.
This is exactly what is done already. The problem really appears to be the timing of cancelling and starting the new job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617295296
> An option would of course be to cancel the active run, but that's a separate question I guess.
This is exactly what is done already. The problem really appears to be the timing of cancelling and starting the new job. Ie the active run is cancelled after the new one is started, or at least that is the order of events that Github sees.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r418364563
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -994,26 +991,24 @@ cd_unittest_ubuntu() {
local mxnet_variant=${1:?"This function requires a mxnet variant as the first argument"}
- local nose_cmd="nosetests-3.4"
-
- $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/unittest
- $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/quantization
+ pytest --durations=50 --verbose tests/python/unittest
+ pytest --durations=50 --verbose tests/python/quantization
# https://github.com/apache/incubator-mxnet/issues/11801
# if [[ ${mxnet_variant} = "cpu" ]] || [[ ${mxnet_variant} = "mkl" ]]; then
# integrationtest_ubuntu_cpu_dist_kvstore
# fi
if [[ ${mxnet_variant} = cu* ]]; then
- $nose_cmd $NOSE_TIMER_ARGUMENTS --verbose tests/python/gpu
+ pytest --durations=50 --verbose tests/python/gpu
Review comment:
Why not? The others are not really of interest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r413477951
##########
File path: ci/jenkins/Jenkinsfile_centos_cpu
##########
@@ -21,7 +21,7 @@
// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
// timeout in minutes
-max_time = 180
+max_time = 240
Review comment:
I noticed that windows tests were running into timeout. also, we should have other ways for handling hanging issue than relying on step level timeout.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617946014
The stability of our test suite is really a pain in the butt :(
You're welcome
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411729150
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
+
+ MXNET_MODULE_SEED=4018804151 pytest --log-level=DEBUG --verbose test_module.py
+
+ Output might be:
+
+ [WARNING] **** module-level seed is set: all tests running deterministically ****
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... [DEBUG] np and mx random seeds = 3935862516
+ ok
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Copy the segfaulting-test seed into the command:
+ MXNET_TEST_SEED=1435005594 pytest --log-level=DEBUG --verbose test_module.py:test2
+ Output might be:
+
+ [INFO] np, mx and python random seeds = 2481884723
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Finally reproduce the segfault directly under gdb (might need additional os packages)
+ by editing the bottom of test_module.py to be
+
+ if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.DEBUG)
+ test2()
+
+ MXNET_TEST_SEED=1435005594 gdb -ex r --args python test_module.py
+
+ 4. When finished debugging the segfault, remember to unset any exported MXNET_ seed
+ variables in the environment to return to non-deterministic testing (a good thing).
+ """
+
+ module_seed_str = os.getenv('MXNET_MODULE_SEED')
+ if module_seed_str is None:
+ seed = random.randint(0, 2**31-1)
+ else:
+ seed = int(module_seed_str)
+ logging.warning('*** module-level seed is set: '
+ 'all tests running deterministically ***')
+ print('Setting module np/mx/python random seeds, '
+ 'use MXNET_MODULE_SEED={} to reproduce.'.format(seed))
+
+ random.seed(seed)
+ try:
+ import numpy as np
+ import mxnet as mx
+ np.random.seed(seed)
+ mx.random.seed(seed)
+ except:
+ print('Unable to import numpy/mxnet. Skipping conftest.')
Review comment:
switched all prints to logging.info/warning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411677218
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
+
+ MXNET_MODULE_SEED=4018804151 pytest --log-level=DEBUG --verbose test_module.py
+
+ Output might be:
+
+ [WARNING] **** module-level seed is set: all tests running deterministically ****
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... [DEBUG] np and mx random seeds = 3935862516
+ ok
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Copy the segfaulting-test seed into the command:
+ MXNET_TEST_SEED=1435005594 pytest --log-level=DEBUG --verbose test_module.py:test2
+ Output might be:
+
+ [INFO] np, mx and python random seeds = 2481884723
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Finally reproduce the segfault directly under gdb (might need additional os packages)
+ by editing the bottom of test_module.py to be
+
+ if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.DEBUG)
+ test2()
+
+ MXNET_TEST_SEED=1435005594 gdb -ex r --args python test_module.py
+
+ 4. When finished debugging the segfault, remember to unset any exported MXNET_ seed
+ variables in the environment to return to non-deterministic testing (a good thing).
+ """
+
+ module_seed_str = os.getenv('MXNET_MODULE_SEED')
+ if module_seed_str is None:
+ seed = random.randint(0, 2**31-1)
+ else:
+ seed = int(module_seed_str)
+ logging.warning('*** module-level seed is set: '
+ 'all tests running deterministically ***')
+ print('Setting module np/mx/python random seeds, '
+ 'use MXNET_MODULE_SEED={} to reproduce.'.format(seed))
+
+ random.seed(seed)
+ try:
+ import numpy as np
+ import mxnet as mx
+ np.random.seed(seed)
+ mx.random.seed(seed)
+ except:
+ print('Unable to import numpy/mxnet. Skipping conftest.')
Review comment:
Inconsistent use of print/logging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-618263117
Woohoo!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617262527
The bot is not responsible for the status publishing, the CI is.
An option would of course be to cancel the active run, but that's a separate question I guess.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411679292
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
+
+ MXNET_MODULE_SEED=4018804151 pytest --log-level=DEBUG --verbose test_module.py
+
+ Output might be:
+
+ [WARNING] **** module-level seed is set: all tests running deterministically ****
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... [DEBUG] np and mx random seeds = 3935862516
+ ok
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Copy the segfaulting-test seed into the command:
+ MXNET_TEST_SEED=1435005594 pytest --log-level=DEBUG --verbose test_module.py:test2
+ Output might be:
+
+ [INFO] np, mx and python random seeds = 2481884723
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Finally reproduce the segfault directly under gdb (might need additional os packages)
+ by editing the bottom of test_module.py to be
+
+ if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.DEBUG)
+ test2()
+
+ MXNET_TEST_SEED=1435005594 gdb -ex r --args python test_module.py
+
+ 4. When finished debugging the segfault, remember to unset any exported MXNET_ seed
+ variables in the environment to return to non-deterministic testing (a good thing).
+ """
+
+ module_seed_str = os.getenv('MXNET_MODULE_SEED')
+ if module_seed_str is None:
+ seed = random.randint(0, 2**31-1)
+ else:
+ seed = int(module_seed_str)
+ logging.warning('*** module-level seed is set: '
+ 'all tests running deterministically ***')
+ print('Setting module np/mx/python random seeds, '
+ 'use MXNET_MODULE_SEED={} to reproduce.'.format(seed))
+
+ random.seed(seed)
+ try:
+ import numpy as np
+ import mxnet as mx
+ np.random.seed(seed)
+ mx.random.seed(seed)
+ except:
+ print('Unable to import numpy/mxnet. Skipping conftest.')
+
+ # The MXNET_TEST_SEED environment variable will override MXNET_MODULE_SEED for tests with
+ # the 'with_seed()' decoration. Inform the user of this once here at the module level.
+ if os.getenv('MXNET_TEST_SEED') is not None:
+ logging.warning('*** test-level seed set: all "@with_seed()" '
+ 'tests run deterministically ***')
+
+
+@pytest.hookimpl(tryfirst=True, hookwrapper=True)
+def pytest_runtest_makereport(item, call):
+ """Make test outcome available to fixture.
+
+ https://docs.pytest.org/en/latest/example/simple.html#making-test-result-information-available-in-fixtures
+ """
+ # execute all other hooks to obtain the report object
+ outcome = yield
+ rep = outcome.get_result()
+
+ # set a report attribute for each phase of a call, which can
+ # be "setup", "call", "teardown"
+ setattr(item, "rep_" + rep.when, rep)
+
+
+@pytest.fixture(scope='function', autouse=True)
+def function_scope_seed(request):
+ """A function scope fixture that manages rng seeds.
+
+ This fixture automatically initializes the python, numpy and mxnet random
+ number generators randomly on every test run.
+
+ def test_ok_with_random_data():
+ ...
+
+ To fix the seed used for a test case mark the test function with the
+ desired seed:
+
+ @pytest.mark.seed(1)
+ def test_not_ok_with_random_data():
+ '''This testcase actually works.'''
+ assert 17 == random.randint(0, 100)
+
+ When a test fails, the fixture outputs the seed used. The user can then set
+ the environment variable MXNET_TEST_SEED to the value reported, then rerun
+ the test with:
+
+ pytest --verbose -s <test_module_name.py> -k <failing_test>
+
+ To run a test repeatedly, install pytest-repeat and add the --count argument:
+
+ pip install pytest-repeat
+ pytest --verbose -s <test_module_name.py> -k <failing_test> --count 1000
+
+ """
+
+ seed = request.node.get_closest_marker('seed')
+ env_seed_str = os.getenv('MXNET_TEST_SEED')
+
+ if seed is not None:
+ seed = seed.args[0]
+ assert isinstance(seed, int)
+ elif env_seed_str is not None:
+ seed = int(env_seed_str)
+ else:
+ seed = random.randint(0, 2**31-1)
+
+ random.seed(seed)
+ try:
+ import numpy as np
+ import mxnet as mx
+ post_test_state = np.random.get_state()
Review comment:
what about the mx.random state? Are we unable to save it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617455399
Would you mind not force pushing so I can review the changes as we go?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617867878
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617994912
This does not seem to be a stability issue, but a bug. Same failure again:
```
[2020-04-22T19:21:39.589Z] Traceback (most recent call last):
[2020-04-22T19:21:39.589Z] File "dist_device_sync_kvstore.py", line 130, in <module>
[2020-04-22T19:21:39.589Z] test_sync_push_pull()
[2020-04-22T19:21:39.589Z] File "dist_device_sync_kvstore.py", line 88, in test_sync_push_pull
[2020-04-22T19:21:39.589Z] check_default_keys(kv, my_rank, nworker, nrepeat=3)
[2020-04-22T19:21:39.589Z] File "dist_device_sync_kvstore.py", line 72, in check_default_keys
[2020-04-22T19:21:39.589Z] check_diff_to_scalar(val, num)
[2020-04-22T19:21:39.589Z] File "dist_device_sync_kvstore.py", line 29, in check_diff_to_scalar
[2020-04-22T19:21:39.589Z] assert(np.sum(np.abs((A - x).asnumpy())) == 0), (rank, A.asnumpy(), x)
[2020-04-22T19:21:39.589Z] AssertionError: (None, array([[0.6, 0.6, 0.6],
[2020-04-22T19:21:39.589Z] [0.6, 0.6, 0.6]], dtype=float32), 41.0)
```
@szha
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411694272
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
Review comment:
Yes, it's already logged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617867976
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617258836
> In that case, the successful one might have overridden the in progress state because it happened later.
@marcoabreu That's the point. The bot is responsible for the order of events thus the bot needs to be investigated. The issue does not appear when manually stopping and then retriggering a job on CI. The bot needs to allow time for Github to report the failed state of the old job before starting the new job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617943879
Ok, the difference is due to hanging issue..
[2020-04-22T17:06:49.477Z] AssertionError: (None, array([[0.6, 0.6, 0.6],
[2020-04-22T17:06:49.477Z] [0.6, 0.6, 0.6]], dtype=float32), 41.0)
[2020-04-22T17:06:49.477Z] terminate called without an active exception
[2020-04-22T17:06:49.477Z] terminate called without an active exception
[2020-04-22T17:06:49.477Z] terminate called without an active exception
[2020-04-22T17:06:49.477Z] terminate called without an active exception
Thanks @marcoabreu to retrigger
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617941907
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617943386
@szha there is a bug with the unix-gpu dist kvstore tests
Runtime on master is 6 minutes http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/master/1887/pipeline/426 but more than 1 hour in this PR http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18025/56/pipeline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617941970
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-618058807
Still
```
[2020-04-22T21:11:19.922Z] + cd /work/mxnet/tests/nightly
[2020-04-22T21:11:19.922Z] + ./test_distributed_training-gpu.sh
[2020-04-22T21:11:41.844Z] worker 0 is initialized
[2020-04-22T21:11:41.844Z] worker 0 passed test_gluon_trainer_type
[2020-04-22T21:11:41.844Z] worker 2 is initialized
[2020-04-22T21:11:41.844Z] worker 2 passed test_gluon_trainer_type
[2020-04-22T21:11:41.844Z] worker 1 is initialized
[2020-04-22T21:11:41.844Z] worker 1 passed test_gluon_trainer_type
[2020-04-22T21:11:41.844Z] worker 3 is initialized
[2020-04-22T21:11:41.844Z] worker 3 passed test_gluon_trainer_type
[2020-04-22T21:11:41.844Z] terminate called without an active exception
[2020-04-22T21:11:41.844Z] terminate called without an active exception
[2020-04-22T21:11:41.844Z] terminate called without an active exception
[2020-04-22T21:11:41.844Z] terminate called without an active exception
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616885701
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616937478
The state should not be published by the bot but by CI. It should report in progress if it has been retriggered. Only exception might be if a new run has been requested before the existing one finished. In that case, the successful one might have overridden the in progress state because it happened later.
It should be investigated, but not as part of the bot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616885731
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-617482184
@marcoabreu sure, I will if feasible. The last one was for resolving a conflict. Unfortunately, I'm about to make another force push due to new conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616910362
The state get's updated after the run finishes.
cc @ChaiBapchya who may fix the bot to display "in progress" state instead of failed state in this situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r413464671
##########
File path: ci/jenkins/Jenkinsfile_centos_cpu
##########
@@ -21,7 +21,7 @@
// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
// timeout in minutes
-max_time = 180
+max_time = 240
Review comment:
why increase the timeout? http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-18025/59/pipeline/
Now the jobs hang even longer before termination
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on a change in pull request #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on a change in pull request #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#discussion_r411695505
##########
File path: conftest.py
##########
@@ -0,0 +1,226 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""conftest.py contains configuration for pytest.
+
+Configuration file for tests in tests/ and scripts/ folders.
+
+Note that fixtures of higher-scoped fixtures (such as ``session``) are
+instantiated before lower-scoped fixtures (such as ``function``).
+
+"""
+
+import logging
+import os
+import random
+
+import pytest
+
+
+def pytest_sessionfinish(session, exitstatus):
+ if exitstatus == 5: # Don't fail if no tests were run
+ session.exitstatus = 0
+
+
+# * Random seed setup
+def pytest_configure():
+ """Pytest configuration hook to help reproduce test segfaults
+
+ Sets and outputs rng seeds.
+
+ The segfault-debug procedure on a module called test_module.py is:
+
+ 1. run "pytest --verbose test_module.py". A seg-faulting output might be:
+
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... ok
+ test_module.test2 ... Illegal instruction (core dumped)
+
+ 2. Copy the module-starting seed into the next command, then run:
+
+ MXNET_MODULE_SEED=4018804151 pytest --log-level=DEBUG --verbose test_module.py
+
+ Output might be:
+
+ [WARNING] **** module-level seed is set: all tests running deterministically ****
+ [INFO] np, mx and python random seeds = 4018804151
+ test_module.test1 ... [DEBUG] np and mx random seeds = 3935862516
+ ok
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Copy the segfaulting-test seed into the command:
+ MXNET_TEST_SEED=1435005594 pytest --log-level=DEBUG --verbose test_module.py:test2
+ Output might be:
+
+ [INFO] np, mx and python random seeds = 2481884723
+ test_module.test2 ... [DEBUG] np and mx random seeds = 1435005594
+ Illegal instruction (core dumped)
+
+ 3. Finally reproduce the segfault directly under gdb (might need additional os packages)
+ by editing the bottom of test_module.py to be
+
+ if __name__ == '__main__':
+ logging.getLogger().setLevel(logging.DEBUG)
+ test2()
+
+ MXNET_TEST_SEED=1435005594 gdb -ex r --args python test_module.py
+
+ 4. When finished debugging the segfault, remember to unset any exported MXNET_ seed
+ variables in the environment to return to non-deterministic testing (a good thing).
+ """
+
+ module_seed_str = os.getenv('MXNET_MODULE_SEED')
+ if module_seed_str is None:
+ seed = random.randint(0, 2**31-1)
+ else:
+ seed = int(module_seed_str)
+ logging.warning('*** module-level seed is set: '
+ 'all tests running deterministically ***')
+ print('Setting module np/mx/python random seeds, '
+ 'use MXNET_MODULE_SEED={} to reproduce.'.format(seed))
+
+ random.seed(seed)
+ try:
+ import numpy as np
+ import mxnet as mx
+ np.random.seed(seed)
+ mx.random.seed(seed)
+ except:
+ print('Unable to import numpy/mxnet. Skipping conftest.')
+
+ # The MXNET_TEST_SEED environment variable will override MXNET_MODULE_SEED for tests with
+ # the 'with_seed()' decoration. Inform the user of this once here at the module level.
+ if os.getenv('MXNET_TEST_SEED') is not None:
+ logging.warning('*** test-level seed set: all "@with_seed()" '
+ 'tests run deterministically ***')
+
+
+@pytest.hookimpl(tryfirst=True, hookwrapper=True)
+def pytest_runtest_makereport(item, call):
+ """Make test outcome available to fixture.
+
+ https://docs.pytest.org/en/latest/example/simple.html#making-test-result-information-available-in-fixtures
+ """
+ # execute all other hooks to obtain the report object
+ outcome = yield
+ rep = outcome.get_result()
+
+ # set a report attribute for each phase of a call, which can
+ # be "setup", "call", "teardown"
+ setattr(item, "rep_" + rep.when, rep)
+
+
+@pytest.fixture(scope='function', autouse=True)
+def function_scope_seed(request):
+ """A function scope fixture that manages rng seeds.
+
+ This fixture automatically initializes the python, numpy and mxnet random
+ number generators randomly on every test run.
+
+ def test_ok_with_random_data():
+ ...
+
+ To fix the seed used for a test case mark the test function with the
+ desired seed:
+
+ @pytest.mark.seed(1)
+ def test_not_ok_with_random_data():
+ '''This testcase actually works.'''
+ assert 17 == random.randint(0, 100)
+
+ When a test fails, the fixture outputs the seed used. The user can then set
+ the environment variable MXNET_TEST_SEED to the value reported, then rerun
+ the test with:
+
+ pytest --verbose -s <test_module_name.py> -k <failing_test>
+
+ To run a test repeatedly, install pytest-repeat and add the --count argument:
+
+ pip install pytest-repeat
+ pytest --verbose -s <test_module_name.py> -k <failing_test> --count 1000
+
+ """
+
+ seed = request.node.get_closest_marker('seed')
+ env_seed_str = os.getenv('MXNET_TEST_SEED')
+
+ if seed is not None:
+ seed = seed.args[0]
+ assert isinstance(seed, int)
+ elif env_seed_str is not None:
+ seed = int(env_seed_str)
+ else:
+ seed = random.randint(0, 2**31-1)
+
+ random.seed(seed)
+ try:
+ import numpy as np
+ import mxnet as mx
+ post_test_state = np.random.get_state()
Review comment:
yes, we are unable to save it yet as we don't expose the states in the frontend yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616805222
Just checked a sample pytest run in CI. Love it! Great initiative, Sheng!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #18025: [DEV] switch nose with pytest
Posted by GitBox <gi...@apache.org>.
szha commented on issue #18025:
URL: https://github.com/apache/incubator-mxnet/pull/18025#issuecomment-616903694
I can see the retry job triggered by the CI bot and the job running, but the state doesn't seem to be reflected in the check status on github.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org