You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/06/09 17:16:48 UTC
[GitHub] [tvm-rfcs] Lunderberg opened a new pull request #7: [UnitTests] Parametrized Unit Tests
Lunderberg opened a new pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7
Migrating earlier conversation from https://discuss.tvm.apache.org/t/rfc-parametrized-unit-tests/9946. RFC should be up to date with all changes/conversations that occurred in that thread.
@areusch Let's continue any discussion here, to avoid having multiple places to check for the same RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698489830
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
Review comment:
Updated as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698487297
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
Review comment:
Good point, and changed. The "should" is a stronger statement, and was what I initially intended, but I'd changed my mind given the current impact to CI runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698487683
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
Review comment:
Updated, mostly following your modified wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698487297
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
Review comment:
Good point, and changed. The "should" is a stronger statement, and was what I initially intended, but I'd changed my mind given the current impact to CI runtime.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
Review comment:
Updated, mostly following your modified wording.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
Review comment:
Updated as suggested.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
Review comment:
Updated, but with a comma instead of a parenthetical.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
Review comment:
Updated as suggested.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
Review comment:
Updated as suggested.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
+`dev`, which can be accepted as input by any test function. These
+replace the previous use of `tvm.testing.enabled_targets()`.
+
+```python
+def test_function_old():
+ for target, dev in tvm.testing.enabled_targets():
+ assert(True) # Target-based test
+
+def test_function_new(target, dev):
+ assert(True) # Target-based test
+```
+
+The parametrized values of `target` are read from the environment
+variable `TVM_TEST_TARGETS`, a semicolon-separated list of targets.
+If `TVM_TEST_TARGETS` is not defined, the target list falls back to
+`tvm.testing.DEFAULT_TEST_TARGETS`. All parametrized targets have
+appropriate markers for checking device capability
+(e.g. `@tvm.testing.uses_gpu`). If a platform cannot run a test, it
+is explicitly listed as being skipped.
+
+It is expected both that enabling unit tests across additional targets
+may uncover several unit tests failures, and that some unit tests may
+fail during the early implementation of supporting a new runtime or
+hardware. In these cases, the `@tvm.testing.known_failing_targets`
+decorator can be used. This marks a test with `pytest.xfail`,
+allowing the test suite to pass. This is intended for cases where an
+implementation will be improved in the future.
+
+```python
+@tvm.testing.known_failing_targets("my_newly_implemented_target")
+def test_function(target, dev):
+ # Test fails on new target, but marking as xfail allows CI suite
+ # to pass during development.
+ assert(target != "my_newly_implemented_target")
+```
+
+If a test should be run over a most targets, but isn't applicable for
+some particular targets, the test should be marked with
+`@tvm.testing.exclude_targets`. For example, a test that exercises
+GPU capabilities may wish to be run against all targets except for
+`llvm`.
+
+```python
+@tvm.testing.excluded_targets("llvm")
+def test_gpu_functionality(target, dev):
+ # Test isn't run on llvm, is excluded from the report entirely.
+ assert(target != "llvm")
+```
+
+If a testing should be run over only a specific set of targets and
+devices, the `@tvm.testing.parametrize_targets` decorator can be used.
+It is intended for use where a test is applicable only to a specific
+target, and is inapplicable to any others (e.g. verifying
+target-specific assembly code matches known assembly code). In most
+circumstances, `@tvm.testing.exclude_targets` or
+`@tvm.testing.known_failing_targets` should be used instead. For
+example, a test that verifies vulkan-specific code generation should
+be marked with `@tvm.testing.parametrize_targets("vulkan")`.
+
+```python
+@tvm.testing.parametrize_targets("vulkan")
+def test_vulkan_codegen(target):
+ f = tvm.build(..., target)
+ assembly = f.imported_modules[0].get_source()
+ assert("%v4bool = OpTypeVector %bool 4" in assembly)
+```
+
+The bare decorator `@tvm.testing.parametrize_targets` is maintained
+for backwards compatibility, but is no longer the preferred style.
+
+## Running Test Subsets
+
+Individual python test files are no longer executable outside of the
+pytest framework. To maintain the existing behavior of running the
+tests defined in a particular file, the following change should be
+made.
+
+```python
+# Before
+if __name__=='__main__':
+ test_function_1()
+ test_function_2()
+ ...
+
+# After
+if __name__=='__main__':
+ sys.exit(pytest.main(sys.argv))
Review comment:
Agreed, and that would help with readability. I've added it to my todo list, though it's low on the priority.
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
+`dev`, which can be accepted as input by any test function. These
+replace the previous use of `tvm.testing.enabled_targets()`.
+
+```python
+def test_function_old():
+ for target, dev in tvm.testing.enabled_targets():
+ assert(True) # Target-based test
+
+def test_function_new(target, dev):
+ assert(True) # Target-based test
+```
+
+The parametrized values of `target` are read from the environment
+variable `TVM_TEST_TARGETS`, a semicolon-separated list of targets.
+If `TVM_TEST_TARGETS` is not defined, the target list falls back to
+`tvm.testing.DEFAULT_TEST_TARGETS`. All parametrized targets have
+appropriate markers for checking device capability
+(e.g. `@tvm.testing.uses_gpu`). If a platform cannot run a test, it
+is explicitly listed as being skipped.
+
+It is expected both that enabling unit tests across additional targets
+may uncover several unit tests failures, and that some unit tests may
+fail during the early implementation of supporting a new runtime or
+hardware. In these cases, the `@tvm.testing.known_failing_targets`
+decorator can be used. This marks a test with `pytest.xfail`,
+allowing the test suite to pass. This is intended for cases where an
+implementation will be improved in the future.
+
+```python
+@tvm.testing.known_failing_targets("my_newly_implemented_target")
+def test_function(target, dev):
+ # Test fails on new target, but marking as xfail allows CI suite
+ # to pass during development.
+ assert(target != "my_newly_implemented_target")
+```
+
+If a test should be run over a most targets, but isn't applicable for
+some particular targets, the test should be marked with
+`@tvm.testing.exclude_targets`. For example, a test that exercises
+GPU capabilities may wish to be run against all targets except for
+`llvm`.
+
+```python
+@tvm.testing.excluded_targets("llvm")
+def test_gpu_functionality(target, dev):
+ # Test isn't run on llvm, is excluded from the report entirely.
+ assert(target != "llvm")
+```
+
+If a testing should be run over only a specific set of targets and
+devices, the `@tvm.testing.parametrize_targets` decorator can be used.
+It is intended for use where a test is applicable only to a specific
+target, and is inapplicable to any others (e.g. verifying
+target-specific assembly code matches known assembly code). In most
+circumstances, `@tvm.testing.exclude_targets` or
+`@tvm.testing.known_failing_targets` should be used instead. For
+example, a test that verifies vulkan-specific code generation should
+be marked with `@tvm.testing.parametrize_targets("vulkan")`.
+
+```python
+@tvm.testing.parametrize_targets("vulkan")
+def test_vulkan_codegen(target):
+ f = tvm.build(..., target)
+ assembly = f.imported_modules[0].get_source()
+ assert("%v4bool = OpTypeVector %bool 4" in assembly)
+```
+
+The bare decorator `@tvm.testing.parametrize_targets` is maintained
+for backwards compatibility, but is no longer the preferred style.
+
+## Running Test Subsets
+
+Individual python test files are no longer executable outside of the
+pytest framework. To maintain the existing behavior of running the
+tests defined in a particular file, the following change should be
+made.
+
+```python
+# Before
+if __name__=='__main__':
+ test_function_1()
+ test_function_2()
+ ...
+
+# After
+if __name__=='__main__':
+ sys.exit(pytest.main(sys.argv))
+```
+
+Alternatively, single files, single tests, or single parameterizations
+of tests can be explicitly specified when calling pytest.
+
+```bash
+# Run all tests in a file
+python3 -mpytest path_to_my_test_file.py
+
+# Run all parameterizations of a single test
+python3 -mpytest path_to_my_test_file.py::test_function_name
+
+# Run a single parameterization of a single test. The brackets should
+# contain the parameters as listed in the pytest verbose output.
+python3 -mpytest 'path_to_my_test_file.py::test_function_name[1024]'
+```
+
+
+## Cache-Related Debugging
+
+If test failure is suspected to be due to multiple tests having access
+to the same cached value, the source of the cross-talk can be narrowed
+down with the following steps.
+
+1. Test with `TVM_TEST_DISABLE_CACHE=1`. If the error stops, then the
+ issue is due to some cache-related cross-talk.
+
+2. Reduce the number of parameters being used for a single unit test,
+ overriding the global parameter definition by marking it with
+ `@pytest.mark.parametrize`. If the error stops, then the issue is
+ due to cross-talk between different parametrizations of a single
+ test.
+
+3. Run a single test function using `python3 -mpytest
+ path/to/my/test_file.py::test_my_test_case`. If the error stops,
+ then the issue is due to cross-talk between the failing unit test
+ and some other unit test in the same file.
+
+ 1. If it is due to cross-talk between multiple unit tests, run the
+ failing unit test alongside each other unit test in the same
+ file that makes use of the cached fixture. This is the same
+ command-line as above, but passing multiple test cases as
+ arguments. If the error stops when run with a particular unit
+ test, then that test is the one that is modifying the cached
+ fixture.
+
+4. Run a single test function on its own, with a single
+ parametrization, using `python3 -mpytest
+ path/to/my/test_file.py::test_my_test_case[parameter_value]`. If
+ the error still occurs, and is still avoided by using
+ `TVM_TEST_DISABLE_CACHE=1`, then the error is in
+ `tvm.testing._fixture_cache`.
+
+
+# Reference-level explanation
+[reference-level-explanation]: #reference-level-explanation
+
+Both `tvm.testing.parameter` and `tvm.testing.fixture` are implemented
+on top of `pytest.fixture`. A call to `tvm.testing.parameter` defines
+a fixture that takes specific values. The following two definitions
+of `array_size` are equivalent.
+
+```python
+# With new functionality
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+# With vanilla pytest functionality
+@pytest.fixture(params=[8, 256, 1024])
+def array_size(request):
+ return request.param
+```
+
+The `@tvm.testing.fixture` without any arguments is equivalent to the
+`@pytest.fixture` without any arguments.
+
+```python
+@tvm.testing.fixture
+def test_data(array_size):
+ return np.random.uniform(size=array_size)
+
+@pytest.fixture
+def test_data(array_size):
+ return np.random.uniform(size=array_size)
+```
+
+The `@tvm.testing.fixture(cached_return_value=True)` does not have a
+direct analog in vanilla pytest. While pytest does allow for re-use
+of fixtures between functions, it only ever maintains [a single cached
+value of each
+fixture](https://docs.pytest.org/en/6.2.x/fixture.html#fixture-scopes).
+This works in cases where only a single cached value is required, but
+causes repeated calls to setup code if a test requires multiple
+different cached values. This can be reduced by careful ordering of
+the pytest fixture scopes, but cannot be completely eliminated. The
+different possible cache usage in vanilla pytest, and with
+`tvm.testing.fixture` are shown below.
+
+```python
+# Possible ordering of tests if `target` is defined in a tighter scope
+# than `array_size`. The call to `generate_setup2` is repeated.
+for array_size in array_sizes:
+ setup1 = generate_setup1(array_size)
+ for target in targets:
+ setup2 = generate_setup2(target)
+ run_test(setup1, setup2)
+
+# Possible ordering of tests if `target` is defined in a tighter scope
+# than `array_size`. The call to `generate_setup2` is repeated.
+for target in targets:
+ setup2 = generate_setup2(target)
+ for array_size in array_sizes:
+ setup1 = generate_setup1(array_size)
+ run_test(setup1, setup2)
+
+# Pseudo-code equivalent of `tvm.testing.fixture(cache_return_value=True)`.
+# No repeated calls to setup code.
+cache_setup1 = {}
+cache_setup2 = {}
+for array_size in array_sizes:
+ for target in targets:
+ if array_size in cache_setup1:
+ setup1 = cache_setup1[array_size]
+ else:
+ setup1 = cache_setup1[array] = generate_setup1(array_size)
+
+ if target in cache_setup2:
+ setup2 = cache_setup2[target]
+ else:
+ setup2 = cache_setup2[target] = generate_setup2(target)
+
+ run_test(setup1, setup2)
+
+del cache_setup1
+del cache_setup2
+```
+
+The cache for a fixture defined with `tvm.testing.fixture` is cleared
+after all tests using that fixture are completed, to avoid excessive
+memory usage.
+
+If a test function is marked with `@pytest.mark.parametrize` for a
+parameter that is also defined with `tvm.testing.parameter`, the test
+function uses only the parameters in `@pytest.mark.parametrize`. This
+allows an individual function to override the parameter definitions if
+needed. Any parameter-dependent fixture are also determined based on
+the values in `@pytest.mark.parametrize`.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+- This makes the individual unit tests be more dependent on the test
+ framework and setup. Incorrect setup may result in confusing test
+ results.
+
+- Caching setup between different tests introduces potential
+ cross-talk between tests. While this risk is also present when
+ looping over parameter values, separating cached values out into
+ fixtures hides that potential cross-talk.
+
+# Rationale and alternatives
+[rationale-and-alternatives]: #rationale-and-alternatives
+
+- Option: Explicitly loop over parameter values or
+ `tvm.testing.enabled_parameters` in the test function. (Most common
+ previous usage.)
+
+ - Pros:
+ - Explicit at the definition of a test function.
+
+ - Cons:
+ - Requires opt-in at each test functions.
+ - Doesn't report information on which parameter value(s) failed.
+
+
+- Option: Use `@tvm.testing.parametrize_targets` as a bare fixture.
+ (Previously implemented behavior, less common usage.)
+
+ - Pros:
+ - Explicit at the definition of a test function.
+
+ - Cons:
+ - Requires opt-in at each test function.
+ - Doesn't provide functionality for shared setup.
+
+
+- Option: Pararametrize using `@pytest.mark.parametrize` rather than
+ `tvm.testing.parameter`.
+
+ - Pros:
+ - Would explicitly show the parameter values next to the function
+ it applies to.
+
+ - Cons:
+ - Must be explicitly added at each test function definition.
+ - Extending the parameters that apply across all tests in a
+ file/directory requires updating several locations.
+ - Similar parameters (e.g. 1000 vs 1024 for an array length) would
+ be defined at separate locations, and would then require
+ separate fixture setup.
+
+# Prior art
+[prior-art]: #prior-art
+
+Discuss prior art, both the good and the bad, in relation to this proposal.
Review comment:
Updated with a reference to `pytest.mark.parametrize` and `pytest.fixture`'s caching using the `scope` parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698489600
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
Review comment:
Updated as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-889964431
@areusch Any status changes on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-875834029
@Lunderberg i think that sounds right. i agree #3 is the overlap between our positions. i'd like for others to comment whether it may be too complex to understand in debugging; otherwise i think we can proceed here. to be explicit: we intend to copy only whitelisted types, and not allow `copy.deepcopy` to work on arbitrary objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r658868650
##########
File path: .gitignore
##########
@@ -0,0 +1,2 @@
+# Emacs temporary files
Review comment:
Fair enough, and I can remove it if that is preferable. I try to avoid using the global `.gitignore` in most cases, so that `git add .` is safe to use in general, and not just on my environment. The tvm repo already has this line in its `.gitignore`, so I figured it was good to add here. (Though in retrospect it probably should have been a separate PR.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-858910805
@Lunderberg I think there were two open items from the discuss post:
1. shadowing local vars with module-level vars containing their fixtures
2. caching non-serializable data
on 1, it sounds like you had found a solution, so I think we're resolved there.
on 2, you'd suggested to table the discussion of how to cache non-serializable objects and just cache serializable ones. that sounds great to me.
so then, the real reason I was having this debate with you: i'd like to propose that in all cases (cache hot or cold), we always provide test functions with a brand-new deserialized copy of the data. i think that solves all of my concerns around tests poisoning downstream tests with cached data, and hopefully has not-too-significant performance impact. what are your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r697878666
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
Review comment:
```suggestion
Before you can use a parameter in a test case, you need to register it with `pytest`.
Do this using the `tvm.testing.parameter` function. For example, the following will
```
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
Review comment:
nit: "could be tested"
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
Review comment:
```suggestion
The simplest implementation would be to write a test function that
internally loops over all parameters and throws an exception when the
test fails. However, this does not give full information to a
developer because `pytest` does not necessarily include the parameter
value in the test report (even when it does, the value is in a different place
depending on the way the internal loop is written). A unit-test that fails for all targets
```
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
Review comment:
```suggestion
A fixture can also depend on parameters or other fixtures. This is
```
##########
File path: .gitignore
##########
@@ -0,0 +1,2 @@
+# Emacs temporary files
Review comment:
okay, please remove this from the change then. we can debate it on another RFC
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
Review comment:
```suggestion
The global TVM test configuration contains definitions for `target` and
```
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
Review comment:
```suggestion
as an input (use the same name for the argument as you used above in the
parameter registration). This test will be run once for each value of the
```
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
+`dev`, which can be accepted as input by any test function. These
+replace the previous use of `tvm.testing.enabled_targets()`.
+
+```python
+def test_function_old():
+ for target, dev in tvm.testing.enabled_targets():
+ assert(True) # Target-based test
+
+def test_function_new(target, dev):
+ assert(True) # Target-based test
+```
+
+The parametrized values of `target` are read from the environment
+variable `TVM_TEST_TARGETS`, a semicolon-separated list of targets.
+If `TVM_TEST_TARGETS` is not defined, the target list falls back to
+`tvm.testing.DEFAULT_TEST_TARGETS`. All parametrized targets have
+appropriate markers for checking device capability
+(e.g. `@tvm.testing.uses_gpu`). If a platform cannot run a test, it
+is explicitly listed as being skipped.
+
+It is expected both that enabling unit tests across additional targets
+may uncover several unit tests failures, and that some unit tests may
+fail during the early implementation of supporting a new runtime or
+hardware. In these cases, the `@tvm.testing.known_failing_targets`
+decorator can be used. This marks a test with `pytest.xfail`,
+allowing the test suite to pass. This is intended for cases where an
+implementation will be improved in the future.
+
+```python
+@tvm.testing.known_failing_targets("my_newly_implemented_target")
+def test_function(target, dev):
+ # Test fails on new target, but marking as xfail allows CI suite
+ # to pass during development.
+ assert(target != "my_newly_implemented_target")
+```
+
+If a test should be run over a most targets, but isn't applicable for
+some particular targets, the test should be marked with
+`@tvm.testing.exclude_targets`. For example, a test that exercises
+GPU capabilities may wish to be run against all targets except for
+`llvm`.
+
+```python
+@tvm.testing.excluded_targets("llvm")
+def test_gpu_functionality(target, dev):
+ # Test isn't run on llvm, is excluded from the report entirely.
+ assert(target != "llvm")
+```
+
+If a testing should be run over only a specific set of targets and
+devices, the `@tvm.testing.parametrize_targets` decorator can be used.
+It is intended for use where a test is applicable only to a specific
+target, and is inapplicable to any others (e.g. verifying
+target-specific assembly code matches known assembly code). In most
+circumstances, `@tvm.testing.exclude_targets` or
+`@tvm.testing.known_failing_targets` should be used instead. For
+example, a test that verifies vulkan-specific code generation should
+be marked with `@tvm.testing.parametrize_targets("vulkan")`.
+
+```python
+@tvm.testing.parametrize_targets("vulkan")
+def test_vulkan_codegen(target):
+ f = tvm.build(..., target)
+ assembly = f.imported_modules[0].get_source()
+ assert("%v4bool = OpTypeVector %bool 4" in assembly)
+```
+
+The bare decorator `@tvm.testing.parametrize_targets` is maintained
+for backwards compatibility, but is no longer the preferred style.
+
+## Running Test Subsets
+
+Individual python test files are no longer executable outside of the
+pytest framework. To maintain the existing behavior of running the
+tests defined in a particular file, the following change should be
+made.
+
+```python
+# Before
+if __name__=='__main__':
+ test_function_1()
+ test_function_2()
+ ...
+
+# After
+if __name__=='__main__':
+ sys.exit(pytest.main(sys.argv))
+```
+
+Alternatively, single files, single tests, or single parameterizations
+of tests can be explicitly specified when calling pytest.
+
+```bash
+# Run all tests in a file
+python3 -mpytest path_to_my_test_file.py
+
+# Run all parameterizations of a single test
+python3 -mpytest path_to_my_test_file.py::test_function_name
+
+# Run a single parameterization of a single test. The brackets should
+# contain the parameters as listed in the pytest verbose output.
+python3 -mpytest 'path_to_my_test_file.py::test_function_name[1024]'
+```
+
+
+## Cache-Related Debugging
+
+If test failure is suspected to be due to multiple tests having access
+to the same cached value, the source of the cross-talk can be narrowed
+down with the following steps.
+
+1. Test with `TVM_TEST_DISABLE_CACHE=1`. If the error stops, then the
+ issue is due to some cache-related cross-talk.
+
+2. Reduce the number of parameters being used for a single unit test,
+ overriding the global parameter definition by marking it with
+ `@pytest.mark.parametrize`. If the error stops, then the issue is
+ due to cross-talk between different parametrizations of a single
+ test.
+
+3. Run a single test function using `python3 -mpytest
+ path/to/my/test_file.py::test_my_test_case`. If the error stops,
+ then the issue is due to cross-talk between the failing unit test
+ and some other unit test in the same file.
+
+ 1. If it is due to cross-talk between multiple unit tests, run the
+ failing unit test alongside each other unit test in the same
+ file that makes use of the cached fixture. This is the same
+ command-line as above, but passing multiple test cases as
+ arguments. If the error stops when run with a particular unit
+ test, then that test is the one that is modifying the cached
+ fixture.
+
+4. Run a single test function on its own, with a single
+ parametrization, using `python3 -mpytest
+ path/to/my/test_file.py::test_my_test_case[parameter_value]`. If
+ the error still occurs, and is still avoided by using
+ `TVM_TEST_DISABLE_CACHE=1`, then the error is in
+ `tvm.testing._fixture_cache`.
+
+
+# Reference-level explanation
+[reference-level-explanation]: #reference-level-explanation
+
+Both `tvm.testing.parameter` and `tvm.testing.fixture` are implemented
+on top of `pytest.fixture`. A call to `tvm.testing.parameter` defines
+a fixture that takes specific values. The following two definitions
+of `array_size` are equivalent.
+
+```python
+# With new functionality
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+# With vanilla pytest functionality
+@pytest.fixture(params=[8, 256, 1024])
+def array_size(request):
+ return request.param
+```
+
+The `@tvm.testing.fixture` without any arguments is equivalent to the
+`@pytest.fixture` without any arguments.
+
+```python
+@tvm.testing.fixture
+def test_data(array_size):
+ return np.random.uniform(size=array_size)
+
+@pytest.fixture
+def test_data(array_size):
+ return np.random.uniform(size=array_size)
+```
+
+The `@tvm.testing.fixture(cached_return_value=True)` does not have a
+direct analog in vanilla pytest. While pytest does allow for re-use
+of fixtures between functions, it only ever maintains [a single cached
+value of each
+fixture](https://docs.pytest.org/en/6.2.x/fixture.html#fixture-scopes).
+This works in cases where only a single cached value is required, but
+causes repeated calls to setup code if a test requires multiple
+different cached values. This can be reduced by careful ordering of
+the pytest fixture scopes, but cannot be completely eliminated. The
+different possible cache usage in vanilla pytest, and with
+`tvm.testing.fixture` are shown below.
+
+```python
+# Possible ordering of tests if `target` is defined in a tighter scope
+# than `array_size`. The call to `generate_setup2` is repeated.
+for array_size in array_sizes:
+ setup1 = generate_setup1(array_size)
+ for target in targets:
+ setup2 = generate_setup2(target)
+ run_test(setup1, setup2)
+
+# Possible ordering of tests if `target` is defined in a tighter scope
+# than `array_size`. The call to `generate_setup2` is repeated.
+for target in targets:
+ setup2 = generate_setup2(target)
+ for array_size in array_sizes:
+ setup1 = generate_setup1(array_size)
+ run_test(setup1, setup2)
+
+# Pseudo-code equivalent of `tvm.testing.fixture(cache_return_value=True)`.
+# No repeated calls to setup code.
+cache_setup1 = {}
+cache_setup2 = {}
+for array_size in array_sizes:
+ for target in targets:
+ if array_size in cache_setup1:
+ setup1 = cache_setup1[array_size]
+ else:
+ setup1 = cache_setup1[array] = generate_setup1(array_size)
+
+ if target in cache_setup2:
+ setup2 = cache_setup2[target]
+ else:
+ setup2 = cache_setup2[target] = generate_setup2(target)
+
+ run_test(setup1, setup2)
+
+del cache_setup1
+del cache_setup2
+```
+
+The cache for a fixture defined with `tvm.testing.fixture` is cleared
+after all tests using that fixture are completed, to avoid excessive
+memory usage.
+
+If a test function is marked with `@pytest.mark.parametrize` for a
+parameter that is also defined with `tvm.testing.parameter`, the test
+function uses only the parameters in `@pytest.mark.parametrize`. This
+allows an individual function to override the parameter definitions if
+needed. Any parameter-dependent fixture are also determined based on
+the values in `@pytest.mark.parametrize`.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+- This makes the individual unit tests be more dependent on the test
+ framework and setup. Incorrect setup may result in confusing test
+ results.
+
+- Caching setup between different tests introduces potential
+ cross-talk between tests. While this risk is also present when
+ looping over parameter values, separating cached values out into
+ fixtures hides that potential cross-talk.
+
+# Rationale and alternatives
+[rationale-and-alternatives]: #rationale-and-alternatives
+
+- Option: Explicitly loop over parameter values or
+ `tvm.testing.enabled_parameters` in the test function. (Most common
+ previous usage.)
+
+ - Pros:
+ - Explicit at the definition of a test function.
+
+ - Cons:
+ - Requires opt-in at each test functions.
+ - Doesn't report information on which parameter value(s) failed.
+
+
+- Option: Use `@tvm.testing.parametrize_targets` as a bare fixture.
+ (Previously implemented behavior, less common usage.)
+
+ - Pros:
+ - Explicit at the definition of a test function.
+
+ - Cons:
+ - Requires opt-in at each test function.
+ - Doesn't provide functionality for shared setup.
+
+
+- Option: Pararametrize using `@pytest.mark.parametrize` rather than
+ `tvm.testing.parameter`.
+
+ - Pros:
+ - Would explicitly show the parameter values next to the function
+ it applies to.
+
+ - Cons:
+ - Must be explicitly added at each test function definition.
+ - Extending the parameters that apply across all tests in a
+ file/directory requires updating several locations.
+ - Similar parameters (e.g. 1000 vs 1024 for an array length) would
+ be defined at separate locations, and would then require
+ separate fixture setup.
+
+# Prior art
+[prior-art]: #prior-art
+
+Discuss prior art, both the good and the bad, in relation to this proposal.
Review comment:
can you remove the boilerplate and just add a reference to i guess `pytest.mark.parametrize` and `functools.lru_cache` or whatever that is now?
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
+`dev`, which can be accepted as input by any test function. These
+replace the previous use of `tvm.testing.enabled_targets()`.
+
+```python
+def test_function_old():
+ for target, dev in tvm.testing.enabled_targets():
+ assert(True) # Target-based test
+
+def test_function_new(target, dev):
+ assert(True) # Target-based test
+```
+
+The parametrized values of `target` are read from the environment
+variable `TVM_TEST_TARGETS`, a semicolon-separated list of targets.
+If `TVM_TEST_TARGETS` is not defined, the target list falls back to
+`tvm.testing.DEFAULT_TEST_TARGETS`. All parametrized targets have
+appropriate markers for checking device capability
+(e.g. `@tvm.testing.uses_gpu`). If a platform cannot run a test, it
+is explicitly listed as being skipped.
+
+It is expected both that enabling unit tests across additional targets
+may uncover several unit tests failures, and that some unit tests may
+fail during the early implementation of supporting a new runtime or
+hardware. In these cases, the `@tvm.testing.known_failing_targets`
+decorator can be used. This marks a test with `pytest.xfail`,
+allowing the test suite to pass. This is intended for cases where an
+implementation will be improved in the future.
+
+```python
+@tvm.testing.known_failing_targets("my_newly_implemented_target")
+def test_function(target, dev):
+ # Test fails on new target, but marking as xfail allows CI suite
+ # to pass during development.
+ assert(target != "my_newly_implemented_target")
+```
+
+If a test should be run over a most targets, but isn't applicable for
+some particular targets, the test should be marked with
+`@tvm.testing.exclude_targets`. For example, a test that exercises
+GPU capabilities may wish to be run against all targets except for
+`llvm`.
+
+```python
+@tvm.testing.excluded_targets("llvm")
+def test_gpu_functionality(target, dev):
+ # Test isn't run on llvm, is excluded from the report entirely.
+ assert(target != "llvm")
+```
+
+If a testing should be run over only a specific set of targets and
+devices, the `@tvm.testing.parametrize_targets` decorator can be used.
+It is intended for use where a test is applicable only to a specific
+target, and is inapplicable to any others (e.g. verifying
+target-specific assembly code matches known assembly code). In most
+circumstances, `@tvm.testing.exclude_targets` or
+`@tvm.testing.known_failing_targets` should be used instead. For
+example, a test that verifies vulkan-specific code generation should
+be marked with `@tvm.testing.parametrize_targets("vulkan")`.
+
+```python
+@tvm.testing.parametrize_targets("vulkan")
+def test_vulkan_codegen(target):
+ f = tvm.build(..., target)
+ assembly = f.imported_modules[0].get_source()
+ assert("%v4bool = OpTypeVector %bool 4" in assembly)
+```
+
+The bare decorator `@tvm.testing.parametrize_targets` is maintained
+for backwards compatibility, but is no longer the preferred style.
+
+## Running Test Subsets
+
+Individual python test files are no longer executable outside of the
+pytest framework. To maintain the existing behavior of running the
+tests defined in a particular file, the following change should be
+made.
+
+```python
+# Before
+if __name__=='__main__':
+ test_function_1()
+ test_function_2()
+ ...
+
+# After
+if __name__=='__main__':
+ sys.exit(pytest.main(sys.argv))
Review comment:
we should ideally split this out into a helper function and eventually add a pylint plugin to enforce, but good to document standard right now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-859671124
@Lunderberg I was thinking you could just require tests provide `bytes` to be cached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch merged pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch merged pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698499102
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
+`dev`, which can be accepted as input by any test function. These
+replace the previous use of `tvm.testing.enabled_targets()`.
+
+```python
+def test_function_old():
+ for target, dev in tvm.testing.enabled_targets():
+ assert(True) # Target-based test
+
+def test_function_new(target, dev):
+ assert(True) # Target-based test
+```
+
+The parametrized values of `target` are read from the environment
+variable `TVM_TEST_TARGETS`, a semicolon-separated list of targets.
+If `TVM_TEST_TARGETS` is not defined, the target list falls back to
+`tvm.testing.DEFAULT_TEST_TARGETS`. All parametrized targets have
+appropriate markers for checking device capability
+(e.g. `@tvm.testing.uses_gpu`). If a platform cannot run a test, it
+is explicitly listed as being skipped.
+
+It is expected both that enabling unit tests across additional targets
+may uncover several unit tests failures, and that some unit tests may
+fail during the early implementation of supporting a new runtime or
+hardware. In these cases, the `@tvm.testing.known_failing_targets`
+decorator can be used. This marks a test with `pytest.xfail`,
+allowing the test suite to pass. This is intended for cases where an
+implementation will be improved in the future.
+
+```python
+@tvm.testing.known_failing_targets("my_newly_implemented_target")
+def test_function(target, dev):
+ # Test fails on new target, but marking as xfail allows CI suite
+ # to pass during development.
+ assert(target != "my_newly_implemented_target")
+```
+
+If a test should be run over a most targets, but isn't applicable for
+some particular targets, the test should be marked with
+`@tvm.testing.exclude_targets`. For example, a test that exercises
+GPU capabilities may wish to be run against all targets except for
+`llvm`.
+
+```python
+@tvm.testing.excluded_targets("llvm")
+def test_gpu_functionality(target, dev):
+ # Test isn't run on llvm, is excluded from the report entirely.
+ assert(target != "llvm")
+```
+
+If a testing should be run over only a specific set of targets and
+devices, the `@tvm.testing.parametrize_targets` decorator can be used.
+It is intended for use where a test is applicable only to a specific
+target, and is inapplicable to any others (e.g. verifying
+target-specific assembly code matches known assembly code). In most
+circumstances, `@tvm.testing.exclude_targets` or
+`@tvm.testing.known_failing_targets` should be used instead. For
+example, a test that verifies vulkan-specific code generation should
+be marked with `@tvm.testing.parametrize_targets("vulkan")`.
+
+```python
+@tvm.testing.parametrize_targets("vulkan")
+def test_vulkan_codegen(target):
+ f = tvm.build(..., target)
+ assembly = f.imported_modules[0].get_source()
+ assert("%v4bool = OpTypeVector %bool 4" in assembly)
+```
+
+The bare decorator `@tvm.testing.parametrize_targets` is maintained
+for backwards compatibility, but is no longer the preferred style.
+
+## Running Test Subsets
+
+Individual python test files are no longer executable outside of the
+pytest framework. To maintain the existing behavior of running the
+tests defined in a particular file, the following change should be
+made.
+
+```python
+# Before
+if __name__=='__main__':
+ test_function_1()
+ test_function_2()
+ ...
+
+# After
+if __name__=='__main__':
+ sys.exit(pytest.main(sys.argv))
+```
+
+Alternatively, single files, single tests, or single parameterizations
+of tests can be explicitly specified when calling pytest.
+
+```bash
+# Run all tests in a file
+python3 -mpytest path_to_my_test_file.py
+
+# Run all parameterizations of a single test
+python3 -mpytest path_to_my_test_file.py::test_function_name
+
+# Run a single parameterization of a single test. The brackets should
+# contain the parameters as listed in the pytest verbose output.
+python3 -mpytest 'path_to_my_test_file.py::test_function_name[1024]'
+```
+
+
+## Cache-Related Debugging
+
+If test failure is suspected to be due to multiple tests having access
+to the same cached value, the source of the cross-talk can be narrowed
+down with the following steps.
+
+1. Test with `TVM_TEST_DISABLE_CACHE=1`. If the error stops, then the
+ issue is due to some cache-related cross-talk.
+
+2. Reduce the number of parameters being used for a single unit test,
+ overriding the global parameter definition by marking it with
+ `@pytest.mark.parametrize`. If the error stops, then the issue is
+ due to cross-talk between different parametrizations of a single
+ test.
+
+3. Run a single test function using `python3 -mpytest
+ path/to/my/test_file.py::test_my_test_case`. If the error stops,
+ then the issue is due to cross-talk between the failing unit test
+ and some other unit test in the same file.
+
+ 1. If it is due to cross-talk between multiple unit tests, run the
+ failing unit test alongside each other unit test in the same
+ file that makes use of the cached fixture. This is the same
+ command-line as above, but passing multiple test cases as
+ arguments. If the error stops when run with a particular unit
+ test, then that test is the one that is modifying the cached
+ fixture.
+
+4. Run a single test function on its own, with a single
+ parametrization, using `python3 -mpytest
+ path/to/my/test_file.py::test_my_test_case[parameter_value]`. If
+ the error still occurs, and is still avoided by using
+ `TVM_TEST_DISABLE_CACHE=1`, then the error is in
+ `tvm.testing._fixture_cache`.
+
+
+# Reference-level explanation
+[reference-level-explanation]: #reference-level-explanation
+
+Both `tvm.testing.parameter` and `tvm.testing.fixture` are implemented
+on top of `pytest.fixture`. A call to `tvm.testing.parameter` defines
+a fixture that takes specific values. The following two definitions
+of `array_size` are equivalent.
+
+```python
+# With new functionality
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+# With vanilla pytest functionality
+@pytest.fixture(params=[8, 256, 1024])
+def array_size(request):
+ return request.param
+```
+
+The `@tvm.testing.fixture` without any arguments is equivalent to the
+`@pytest.fixture` without any arguments.
+
+```python
+@tvm.testing.fixture
+def test_data(array_size):
+ return np.random.uniform(size=array_size)
+
+@pytest.fixture
+def test_data(array_size):
+ return np.random.uniform(size=array_size)
+```
+
+The `@tvm.testing.fixture(cached_return_value=True)` does not have a
+direct analog in vanilla pytest. While pytest does allow for re-use
+of fixtures between functions, it only ever maintains [a single cached
+value of each
+fixture](https://docs.pytest.org/en/6.2.x/fixture.html#fixture-scopes).
+This works in cases where only a single cached value is required, but
+causes repeated calls to setup code if a test requires multiple
+different cached values. This can be reduced by careful ordering of
+the pytest fixture scopes, but cannot be completely eliminated. The
+different possible cache usage in vanilla pytest, and with
+`tvm.testing.fixture` are shown below.
+
+```python
+# Possible ordering of tests if `target` is defined in a tighter scope
+# than `array_size`. The call to `generate_setup2` is repeated.
+for array_size in array_sizes:
+ setup1 = generate_setup1(array_size)
+ for target in targets:
+ setup2 = generate_setup2(target)
+ run_test(setup1, setup2)
+
+# Possible ordering of tests if `target` is defined in a tighter scope
+# than `array_size`. The call to `generate_setup2` is repeated.
+for target in targets:
+ setup2 = generate_setup2(target)
+ for array_size in array_sizes:
+ setup1 = generate_setup1(array_size)
+ run_test(setup1, setup2)
+
+# Pseudo-code equivalent of `tvm.testing.fixture(cache_return_value=True)`.
+# No repeated calls to setup code.
+cache_setup1 = {}
+cache_setup2 = {}
+for array_size in array_sizes:
+ for target in targets:
+ if array_size in cache_setup1:
+ setup1 = cache_setup1[array_size]
+ else:
+ setup1 = cache_setup1[array] = generate_setup1(array_size)
+
+ if target in cache_setup2:
+ setup2 = cache_setup2[target]
+ else:
+ setup2 = cache_setup2[target] = generate_setup2(target)
+
+ run_test(setup1, setup2)
+
+del cache_setup1
+del cache_setup2
+```
+
+The cache for a fixture defined with `tvm.testing.fixture` is cleared
+after all tests using that fixture are completed, to avoid excessive
+memory usage.
+
+If a test function is marked with `@pytest.mark.parametrize` for a
+parameter that is also defined with `tvm.testing.parameter`, the test
+function uses only the parameters in `@pytest.mark.parametrize`. This
+allows an individual function to override the parameter definitions if
+needed. Any parameter-dependent fixture are also determined based on
+the values in `@pytest.mark.parametrize`.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+- This makes the individual unit tests be more dependent on the test
+ framework and setup. Incorrect setup may result in confusing test
+ results.
+
+- Caching setup between different tests introduces potential
+ cross-talk between tests. While this risk is also present when
+ looping over parameter values, separating cached values out into
+ fixtures hides that potential cross-talk.
+
+# Rationale and alternatives
+[rationale-and-alternatives]: #rationale-and-alternatives
+
+- Option: Explicitly loop over parameter values or
+ `tvm.testing.enabled_parameters` in the test function. (Most common
+ previous usage.)
+
+ - Pros:
+ - Explicit at the definition of a test function.
+
+ - Cons:
+ - Requires opt-in at each test functions.
+ - Doesn't report information on which parameter value(s) failed.
+
+
+- Option: Use `@tvm.testing.parametrize_targets` as a bare fixture.
+ (Previously implemented behavior, less common usage.)
+
+ - Pros:
+ - Explicit at the definition of a test function.
+
+ - Cons:
+ - Requires opt-in at each test function.
+ - Doesn't provide functionality for shared setup.
+
+
+- Option: Pararametrize using `@pytest.mark.parametrize` rather than
+ `tvm.testing.parameter`.
+
+ - Pros:
+ - Would explicitly show the parameter values next to the function
+ it applies to.
+
+ - Cons:
+ - Must be explicitly added at each test function definition.
+ - Extending the parameters that apply across all tests in a
+ file/directory requires updating several locations.
+ - Similar parameters (e.g. 1000 vs 1024 for an array length) would
+ be defined at separate locations, and would then require
+ separate fixture setup.
+
+# Prior art
+[prior-art]: #prior-art
+
+Discuss prior art, both the good and the bad, in relation to this proposal.
Review comment:
Updated with a reference to `pytest.mark.parametrize` and `pytest.fixture`'s caching using the `scope` parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-859664274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch merged pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch merged pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r658856543
##########
File path: .gitignore
##########
@@ -0,0 +1,2 @@
+# Emacs temporary files
Review comment:
i think these should go in your global .gitignore not the project-specific one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698489224
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
Review comment:
Updated, but with a comma instead of a parenthetical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698488460
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
Review comment:
Updated as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-864285376
@areusch I've added verbiage to the RFC to indicate use of `copy.deepcopy` for the results of cached fixtures, and have updated the [PR](https://github.com/apache/tvm/pull/8010) with the copying. The error message if a non-serializable type is cached gives a link to this RFC, to avoid any confusion in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#issuecomment-870833178
Copying some notes from a 1:1, along with commentary at the bottom.
Caching options listed below, arranged from most flexibility at the top to least flexibility at the bottom.
1. All caching allowed. A fixture tagged with `cache_return_value=True` can return any python object. All tests that use the fixture receive the same python object as input.
- Pros
- Most potential to speed up running of CI.
- Most potential to speed up by cross-function/file use of shared setup (e.g. np.random.uniform)
- Works for any type.
- Cons
- Most potential to have interacting unit tests.
- Could make somebody think that interactions are allowed between unit tests.
2. Cache only copy-able types. A fixture tagged with `cache_return_value=True` has cached return values, but each parametrized unit test receives a copy made by `copy.deepcopy` prior to returning the object. Any types that raise an error during copying result in a failed setup for the test.
- Pros
- Very simple to add caching to existing functions.
- Cons
- Unintentionally copy global state that is pointed to by local state. Doesn't impact correctness, but would increase runtime.
- Less flexibility, an object that cannot be deepcopy-ed results in a type error.
3. Cache only copy-able types, but with a custom implementation of `copy.deepcopy`. Would follow the same rules if a `__deepcopy__` or pair of `__getstate__`/ `__setstate__` exists, but would otherwise require the type to be in a list of allowed types. List of types
- Pros
- Opt-in both by fixture and by data-type.
- Cons
- Re-implements a common utility.
4. Cache using explicitly specified serialize/deserialize functions. A fixture tagged with `cache_return_value=True` must return three values. The first is the fixture value itself. The second/third are serialize/deserialize functions for that value.
- Pros
- Cons
5. Only caching of `bytes` type is allowed. A fixture tagged with `cache_return_value=True` must return a `bytes` object. This object is deserialized at the start of the test function into the desired type.
- Pros
- Cons
6. No caching. All fixtures are recomputed for each parametrized value.
- Pros
- All unit tests are independent by design. Zero interaction possible.
- Cons
- Most potential impact to CI runtime.
Overall, @areusch 's concern is avoiding hidden failure modes, especially where there could be tests that only appear to pass due to cross-talk. As such, he is more comfortable with more explicit usage. and would be comfortable with any of implementations 3-5. @Lunderberg 's main concern is usability and ease of enabling, such that a slow test can have caching enabled with minimal effort where it is needed, and would be comfortable with any of implementations 2-3. One key point that we came up with is that we expect fixtures to be uncached by default (`@tvm.testing.fixture`), and to have caching enabled on a per-fixture basis (`@tvm.testing.fixture(cache_return_value=True)` where there are performance benefits to be gained.
Since #3 is the overlap we both like between having opt-in caching, both on a per-return-value and per-fixture basis, I've put together [a test implementation](https://gist.github.com/Lunderberg/039f2f1d14c9ad7de88e1d2224a430cc). It calls `copy.deepcopy`, but passes in a custom `memo` argument that performs type-checking of the copied type. Types can be explicitly listed as being safe to copy, but the use of the default `object.__reduce__` is disabled. This implementation minimizes the amount of re-implementing that would need to be done, but does rely on `ctypes` to identify the object being checked.
@areusch Can you check this over to make sure I've given an accurate description of your comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] areusch merged pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
areusch merged pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm-rfcs] Lunderberg commented on a change in pull request #7: [UnitTests] Parametrized Unit Tests
Posted by GitBox <gi...@apache.org>.
Lunderberg commented on a change in pull request #7:
URL: https://github.com/apache/tvm-rfcs/pull/7#discussion_r698491103
##########
File path: rfcs/0007-parametrized-unit-tests.md
##########
@@ -0,0 +1,568 @@
+- Feature Name: Parametrized Unit Tests
+- Start Date: 2021-05-10(fill me in with today's date, YYYY-MM-DD)
+- RFC PR: [apache/tvm-rfcs#0007](https://github.com/apache/tvm-rfcs/pull/0007)
+- GitHub PR: [apache/tvm#8010](https://github.com/apache/tvm/issues/8010)
+
+# Summary
+[summary]: #summary
+
+This RFC documents how to implement unit tests that depend on input
+parameters, or have setup that depends on input parameters.
+
+# Motivation
+[motivation]: #motivation
+
+Some unit tests should be tested along a variety of parameters for
+better coverage. For example, a unit test that does not depend on
+target-specific features should be tested on all targets that the test
+platform supports. Alternatively, a unit test may need to pass
+different array sizes to a function, in order to exercise different
+code paths within that function.
+
+The simplest implementation would be to write a test function that
+loops over all parameters, throwing an exception if any parameter
+fails the test. However, this does not give full information to a
+developer, as a failure from any parameter results in the entire test
+to be marked as failing. A unit-test that fails for all targets
+requires different debugging than a unit-test that fails on a single
+specific target, and so this information should be exposed.
+
+This RFC adds functionality for implementing parameterized unit tests,
+such that each set of parameters appears as a separate test result in
+the final output.
+
+# Guide-level explanation
+[guide-level-explanation]: #guide-level-explanation
+
+## Parameters
+
+To make a new parameter for unit tests to use, define it with the
+`tvm.testing.parameter` function. For example, the following will
+define a parameter named `array_size` that has three possible values.
+This can appear either at global scope inside a test module to be
+usable by all test functions in that module, or in a directory's
+`conftest.py` to be usable by all tests in that directory.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+```
+
+To use a parameter, define a test function that accepts the parameter
+as an input. This test will be run once for each value of the
+parameter. For example, the `test_function` below would be run three
+times, each time with a different value of `array_size` according to
+the earlier definition. These would show up in the output report as
+`test_function[8]`, `test_function[256]`, and `test_function[1024]`,
+with the name of the parameter as part of the function.
+
+```python
+def test_function(array_size):
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+If a parameter is used by a test function, but isn't declared as a
+function argument, it will produce a `NameError` when accessed. This
+happens even if the parameter is defined at module scope, and would
+otherwise be accessible by the usual scoping rules. This is
+intentional, as access of the global variable would otherwise access
+an `array_size` function definition, rather than the specific
+parameter value.
+
+```python
+def test_function_broken():
+ # Throws NameError, undefined variable "array_size"
+ input_array = np.random.uniform(size=array_size)
+ # Test code here
+```
+
+By default, a test function that accepts multiple parameters as
+arguments will be run for all combinations of values of those
+parameters. If only some combinations of parameters should be used,
+the `tvm.testing.parameters` function can be used to simultaneously
+define multiple parameters. A test function that accepts parameters
+that were defined through `tvm.testing.parameters` will only be called
+once for each set of parameters.
+
+```python
+array_size = tvm.testing.parameter(8, 256, 1024)
+dtype = tvm.testing.parameter('float32', 'int32')
+
+# Called 6 times, once for each combination of array_size and dtype.
+def test_function1(array_size, dtype):
+ assert(True)
+
+test_data, reference_result = tvm.testing.parameters(
+ ('test_data_1.dat', 'result_1.txt'),
+ ('test_data_2.dat', 'result_2.txt'),
+ ('test_data_3.dat', 'result_3.txt'),
+)
+
+# Called 3 times, once for each (test_data, reference_result) tuple.
+def test_function3(test_data, reference_result):
+ assert(True)
+```
+
+## Fixtures
+
+Fixtures in pytest separate setup code from test code, and are used
+for two primary purposes. The first is for improved readability when
+debugging, so that a failure in the setup is distinguishable from a
+failure in the test. The second is to avoid performing expensive test
+setup that is shared across multiple tests, letting the test suite run
+faster.
+
+For example, the following function first reads test data, and then
+performs tests that use the test data.
+
+```python
+# test_function_old() calls read_test_data(). If read_test_data()
+# throws an error, test_function_old() shows as a failed test.
+
+def test_function_old():
+ dataset = read_test_data()
+ assert(True) # Test succeeds
+```
+
+This can be pulled out into a separate setup function, which the test
+function then accepts as an argument. In this usage, this is
+equivalent to using a bare `@pytest.fixture` decorator. By default,
+the fixture value is recalculated for every test function, to minimize
+the potential for interaction between unit tests.
+
+```python
+@tvm.testing.fixture
+def dataset():
+ print('Prints once for each test function that uses dataset.')
+ return read_test_data()
+
+# test_function_new() accepts the dataset fixture. If
+# read_test_data() throws an error, test_function_new() shows
+# as unrunnable.
+def test_function_new(dataset):
+ assert(True) # Test succeeds
+```
+
+If the fixture is more expensive to calculate, then it may be worth
+caching the computed fixture. This is done with the
+`cache_return_value=True` argument.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ print('Prints once no matter how many test functions use dataset.')
+ return download_test_data()
+
+def test_function(dataset):
+ assert(True) # Test succeeds
+```
+
+The caching can be disabled entirely by setting the environment
+variable `TVM_TEST_DISABLE_CACHE` to a non-zero integer. This can be
+useful to re-run tests that failed, to check whether the failure is
+due to modification/re-use of a cached value.
+
+A fixture can depend on parameters, or on other fixtures. This is
+defined by accepting additional parameters. For example, consider the
+following test function. In this example, the calculation of
+`correct_output` depends on the test data, and the `schedule` depends
+on some block size. The `generate_output` function contains the
+functionality to be tested.
+
+```python
+def test_function_old():
+ dataset = download_test_data()
+ correct_output = calculate_correct_output(dataset)
+ for block_size in [8, 256, 1024]:
+ schedule = setup_schedule(block_size)
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+These can be split out into separate parameters and fixtures to
+isolate the functionality to be tested. Whether to split out the
+setup code, and whether to cache it is dependent on the test function,
+how expensive the setup is to perform the setup, whether other tests
+can share the same setup code, and so on.
+
+```python
+@tvm.testing.fixture(cache_return_value = True)
+def dataset():
+ return download_test_data()
+
+@tvm.testing.fixture
+def correct_output(dataset):
+ return calculate_correct_output(dataset)
+
+array_size = tvm.testing.parameter(8, 256, 1024)
+
+@tvm.testing.fixture
+def schedule(array_size):
+ return setup_schedule(array_size)
+
+def test_function_new(dataset, correct_output, schedule):
+ output = generate_output(dataset, schedule)
+ tvm.testing.assert_allclose(output, correct_output)
+```
+
+## Target/Device Parametrization
+
+The TVM test configuration contains definitions for `target` and
+`dev`, which can be accepted as input by any test function. These
+replace the previous use of `tvm.testing.enabled_targets()`.
+
+```python
+def test_function_old():
+ for target, dev in tvm.testing.enabled_targets():
+ assert(True) # Target-based test
+
+def test_function_new(target, dev):
+ assert(True) # Target-based test
+```
+
+The parametrized values of `target` are read from the environment
+variable `TVM_TEST_TARGETS`, a semicolon-separated list of targets.
+If `TVM_TEST_TARGETS` is not defined, the target list falls back to
+`tvm.testing.DEFAULT_TEST_TARGETS`. All parametrized targets have
+appropriate markers for checking device capability
+(e.g. `@tvm.testing.uses_gpu`). If a platform cannot run a test, it
+is explicitly listed as being skipped.
+
+It is expected both that enabling unit tests across additional targets
+may uncover several unit tests failures, and that some unit tests may
+fail during the early implementation of supporting a new runtime or
+hardware. In these cases, the `@tvm.testing.known_failing_targets`
+decorator can be used. This marks a test with `pytest.xfail`,
+allowing the test suite to pass. This is intended for cases where an
+implementation will be improved in the future.
+
+```python
+@tvm.testing.known_failing_targets("my_newly_implemented_target")
+def test_function(target, dev):
+ # Test fails on new target, but marking as xfail allows CI suite
+ # to pass during development.
+ assert(target != "my_newly_implemented_target")
+```
+
+If a test should be run over a most targets, but isn't applicable for
+some particular targets, the test should be marked with
+`@tvm.testing.exclude_targets`. For example, a test that exercises
+GPU capabilities may wish to be run against all targets except for
+`llvm`.
+
+```python
+@tvm.testing.excluded_targets("llvm")
+def test_gpu_functionality(target, dev):
+ # Test isn't run on llvm, is excluded from the report entirely.
+ assert(target != "llvm")
+```
+
+If a testing should be run over only a specific set of targets and
+devices, the `@tvm.testing.parametrize_targets` decorator can be used.
+It is intended for use where a test is applicable only to a specific
+target, and is inapplicable to any others (e.g. verifying
+target-specific assembly code matches known assembly code). In most
+circumstances, `@tvm.testing.exclude_targets` or
+`@tvm.testing.known_failing_targets` should be used instead. For
+example, a test that verifies vulkan-specific code generation should
+be marked with `@tvm.testing.parametrize_targets("vulkan")`.
+
+```python
+@tvm.testing.parametrize_targets("vulkan")
+def test_vulkan_codegen(target):
+ f = tvm.build(..., target)
+ assembly = f.imported_modules[0].get_source()
+ assert("%v4bool = OpTypeVector %bool 4" in assembly)
+```
+
+The bare decorator `@tvm.testing.parametrize_targets` is maintained
+for backwards compatibility, but is no longer the preferred style.
+
+## Running Test Subsets
+
+Individual python test files are no longer executable outside of the
+pytest framework. To maintain the existing behavior of running the
+tests defined in a particular file, the following change should be
+made.
+
+```python
+# Before
+if __name__=='__main__':
+ test_function_1()
+ test_function_2()
+ ...
+
+# After
+if __name__=='__main__':
+ sys.exit(pytest.main(sys.argv))
Review comment:
Agreed, and that would help with readability. I've added it to my todo list, though it's low on the priority.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org