You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/06/25 18:58:26 UTC
[GitHub] [tvm] Lunderberg opened a new pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Lunderberg opened a new pull request #8343:
URL: https://github.com/apache/tvm/pull/8343
In these cases, the test function should be marked as failing the setup, and should not run. This is pytest's default behavior, and should work whether or not a fixture is cached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#discussion_r660873954
##########
File path: tests/python/unittest/test_tvm_testing_features.py
##########
@@ -145,5 +145,37 @@ def test_cached_count(self):
assert self.cached_calls == len(self.param1_vals)
+class TestBrokenFixture:
+ # Tests that use a fixture that throws an exception fail, and are
+ # marked as setup failures. The tests themselves are never run.
+ # This behavior should be the same whether or not the fixture
+ # results are cached.
+
+ num_uses_broken_uncached_fixture = 0
+ num_uses_broken_cached_fixture = 0
+
+ @tvm.testing.fixture
+ def broken_uncached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_uncached_fixture(self, broken_uncached_fixture):
+ type(self).num_uses_broken_fixture += 1
+
+ def test_num_uses_uncached(self):
+ assert self.num_uses_broken_uncached_fixture == 0
+
+ @tvm.testing.fixture(cache_return_value=True)
+ def broken_cached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_cached_fixture(self, broken_cached_fixture):
+ type(self).num_uses_broken_cached_fixture += 1
+
+ def test_num_uses_cached(self):
Review comment:
this seems like a reasonably easy thing to do. thanks @Lunderberg !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] masahi merged pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
masahi merged pull request #8343:
URL: https://github.com/apache/tvm/pull/8343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Lunderberg commented on a change in pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#discussion_r660038825
##########
File path: tests/python/unittest/test_tvm_testing_features.py
##########
@@ -145,5 +145,37 @@ def test_cached_count(self):
assert self.cached_calls == len(self.param1_vals)
+class TestBrokenFixture:
+ # Tests that use a fixture that throws an exception fail, and are
+ # marked as setup failures. The tests themselves are never run.
+ # This behavior should be the same whether or not the fixture
+ # results are cached.
+
+ num_uses_broken_uncached_fixture = 0
+ num_uses_broken_cached_fixture = 0
+
+ @tvm.testing.fixture
+ def broken_uncached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_uncached_fixture(self, broken_uncached_fixture):
+ type(self).num_uses_broken_fixture += 1
+
+ def test_num_uses_uncached(self):
+ assert self.num_uses_broken_uncached_fixture == 0
+
+ @tvm.testing.fixture(cache_return_value=True)
+ def broken_cached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_cached_fixture(self, broken_cached_fixture):
+ type(self).num_uses_broken_cached_fixture += 1
+
+ def test_num_uses_cached(self):
Review comment:
Currently, it will not. We'd need to also add a call to `pytest_xdist_make_scheduler` ([example stack overflow post](https://stackoverflow.com/a/59504228)) in order to force these tests to be run in order on a single node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Lunderberg commented on a change in pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#discussion_r660838294
##########
File path: tests/python/unittest/test_tvm_testing_features.py
##########
@@ -145,5 +145,37 @@ def test_cached_count(self):
assert self.cached_calls == len(self.param1_vals)
+class TestBrokenFixture:
+ # Tests that use a fixture that throws an exception fail, and are
+ # marked as setup failures. The tests themselves are never run.
+ # This behavior should be the same whether or not the fixture
+ # results are cached.
+
+ num_uses_broken_uncached_fixture = 0
+ num_uses_broken_cached_fixture = 0
+
+ @tvm.testing.fixture
+ def broken_uncached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_uncached_fixture(self, broken_uncached_fixture):
+ type(self).num_uses_broken_fixture += 1
+
+ def test_num_uses_uncached(self):
+ assert self.num_uses_broken_uncached_fixture == 0
+
+ @tvm.testing.fixture(cache_return_value=True)
+ def broken_cached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_cached_fixture(self, broken_cached_fixture):
+ type(self).num_uses_broken_cached_fixture += 1
+
+ def test_num_uses_cached(self):
Review comment:
As mentioned in our conversation yesterday, I've added a comment regarding pytest-xdist and the link to the above stackoverflow post to this PR, so that it will be easier to find if/when we parallelize the testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Lunderberg commented on pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#issuecomment-868770496
@areusch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] Lunderberg commented on a change in pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#discussion_r660038825
##########
File path: tests/python/unittest/test_tvm_testing_features.py
##########
@@ -145,5 +145,37 @@ def test_cached_count(self):
assert self.cached_calls == len(self.param1_vals)
+class TestBrokenFixture:
+ # Tests that use a fixture that throws an exception fail, and are
+ # marked as setup failures. The tests themselves are never run.
+ # This behavior should be the same whether or not the fixture
+ # results are cached.
+
+ num_uses_broken_uncached_fixture = 0
+ num_uses_broken_cached_fixture = 0
+
+ @tvm.testing.fixture
+ def broken_uncached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_uncached_fixture(self, broken_uncached_fixture):
+ type(self).num_uses_broken_fixture += 1
+
+ def test_num_uses_uncached(self):
+ assert self.num_uses_broken_uncached_fixture == 0
+
+ @tvm.testing.fixture(cache_return_value=True)
+ def broken_cached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_cached_fixture(self, broken_cached_fixture):
+ type(self).num_uses_broken_cached_fixture += 1
+
+ def test_num_uses_cached(self):
Review comment:
Currently, it will not. We'd need to also add a call to `pytest_xdist_make_scheduler` ([example stack overflow post](https://stackoverflow.com/a/59504228)) in order to force these tests to be run in order on a single node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#discussion_r659995934
##########
File path: tests/python/unittest/test_tvm_testing_features.py
##########
@@ -145,5 +145,37 @@ def test_cached_count(self):
assert self.cached_calls == len(self.param1_vals)
+class TestBrokenFixture:
+ # Tests that use a fixture that throws an exception fail, and are
+ # marked as setup failures. The tests themselves are never run.
+ # This behavior should be the same whether or not the fixture
+ # results are cached.
+
+ num_uses_broken_uncached_fixture = 0
+ num_uses_broken_cached_fixture = 0
+
+ @tvm.testing.fixture
+ def broken_uncached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_uncached_fixture(self, broken_uncached_fixture):
+ type(self).num_uses_broken_fixture += 1
+
+ def test_num_uses_uncached(self):
+ assert self.num_uses_broken_uncached_fixture == 0
+
+ @tvm.testing.fixture(cache_return_value=True)
+ def broken_cached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_cached_fixture(self, broken_cached_fixture):
+ type(self).num_uses_broken_cached_fixture += 1
+
+ def test_num_uses_cached(self):
Review comment:
if we [parallelize testing](https://pypi.org/project/pytest-xdist/#parallelization), will this inter-test dependency work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #8343: [Unittests] Added a meta-test for tvm.testing.fixture behavior in case of a broken fixture.
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #8343:
URL: https://github.com/apache/tvm/pull/8343#discussion_r659995934
##########
File path: tests/python/unittest/test_tvm_testing_features.py
##########
@@ -145,5 +145,37 @@ def test_cached_count(self):
assert self.cached_calls == len(self.param1_vals)
+class TestBrokenFixture:
+ # Tests that use a fixture that throws an exception fail, and are
+ # marked as setup failures. The tests themselves are never run.
+ # This behavior should be the same whether or not the fixture
+ # results are cached.
+
+ num_uses_broken_uncached_fixture = 0
+ num_uses_broken_cached_fixture = 0
+
+ @tvm.testing.fixture
+ def broken_uncached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_uncached_fixture(self, broken_uncached_fixture):
+ type(self).num_uses_broken_fixture += 1
+
+ def test_num_uses_uncached(self):
+ assert self.num_uses_broken_uncached_fixture == 0
+
+ @tvm.testing.fixture(cache_return_value=True)
+ def broken_cached_fixture(self):
+ raise RuntimeError("Intentionally broken fixture")
+
+ @pytest.mark.xfail(True, reason="Broken fixtures should result in a failing setup", strict=True)
+ def test_uses_broken_cached_fixture(self, broken_cached_fixture):
+ type(self).num_uses_broken_cached_fixture += 1
+
+ def test_num_uses_cached(self):
Review comment:
if we [parallelize testing](https://pypi.org/project/pytest-xdist/#parallelization), will this inter-test dependency work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org