[GitHub] [tvm] mehrdadh commented on a diff in pull request #12207: [microTVM] Refactor pytest fixtures

[GitBox <gi...@apache.org>]

mehrdadh commented on code in PR #12207:
URL: https://github.com/apache/tvm/pull/12207#discussion_r933629167


##########
python/tvm/micro/testing/evaluation.py:
##########
@@ -153,4 +163,4 @@ def evaluate_model_accuracy(session, aot_executor, input_data, true_labels, runs
     num_correct = sum(u == v for u, v in zip(true_labels, predicted_labels))
     average_time = sum(aot_runtimes) / len(aot_runtimes)
     accuracy = num_correct / len(predicted_labels)
-    return average_time, accuracy
+    return average_time, accuracy, predicted_labels

Review Comment:
   I made this change since we were using the predicted labels outside of this function for validity check in some tests. If you think it's out of the scope of this function, I can replicated the whole function in the other repo. thoughts?



##########
python/tvm/micro/testing/pytest_plugin.py:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=invalid-name,redefined-outer-name
+""" microTVM testing fixtures used to deduce testing argument
+    values from testing parameters """
+
+import pathlib
+import json
+import os
+import datetime
+import pytest
+
+import tvm
+from tvm.contrib.utils import tempdir
+
+
+def zephyr_boards() -> dict:
+    """Returns a dict mapping board to target model"""
+    with open(
+        pathlib.Path(tvm.micro.get_microtvm_template_projects("zephyr")) / "boards.json"
+    ) as f:
+        board_properties = json.load(f)
+
+    boards_model = {board: info["model"] for board, info in board_properties.items()}
+    return boards_model
+
+
+def get_boards(platform: str) -> dict:

Review Comment:
   removed get_boards



##########
python/tvm/micro/testing/pytest_plugin.py:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=invalid-name,redefined-outer-name
+""" microTVM testing fixtures used to deduce testing argument
+    values from testing parameters """
+
+import pathlib
+import json
+import os
+import datetime
+import pytest
+
+import tvm
+from tvm.contrib.utils import tempdir
+
+
+def zephyr_boards() -> dict:
+    """Returns a dict mapping board to target model"""
+    with open(
+        pathlib.Path(tvm.micro.get_microtvm_template_projects("zephyr")) / "boards.json"
+    ) as f:
+        board_properties = json.load(f)
+
+    boards_model = {board: info["model"] for board, info in board_properties.items()}
+    return boards_model
+
+
+def get_boards(platform: str) -> dict:
+    """Returns a dict mapping board to target model"""
+    with open(
+        pathlib.Path(tvm.micro.get_microtvm_template_projects(platform)) / "boards.json"
+    ) as f:
+        board_properties = json.load(f)
+
+    boards_model = {board: info["model"] for board, info in board_properties.items()}
+    return boards_model
+
+
+def pytest_addoption(parser):
+    """Adds more pytest arguments"""
+    parser.addoption(
+        "--board",

Review Comment:
   added more details



##########
tests/micro/zephyr/test_zephyr.py:
##########
@@ -89,7 +89,7 @@ def _make_add_sess(temp_dir, model, zephyr_board, west_cmd, build_config, dtype=
 # The same test code can be executed on both the QEMU simulation and on real hardware.
 @tvm.testing.requires_micro
 @pytest.mark.skip_boards(["mps2_an521"])
-def test_add_uint(temp_dir, board, west_cmd, tvm_debug):
+def test_add_uint(workspace_dir, board, west_cmd, tvm_debug):

Review Comment:
   `workspace_dir` is a custom path directory where it is in the tvm workspace, so user can easily access them even when they are testing in docker. Also, we have `tvm-debug` which we can use to keep the project for debugging purposes. I think it's a useful tool for developers.



##########
python/tvm/micro/testing/pytest_plugin.py:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=invalid-name,redefined-outer-name
+""" microTVM testing fixtures used to deduce testing argument
+    values from testing parameters """
+
+import pathlib
+import json
+import os
+import datetime
+import pytest
+
+import tvm
+from tvm.contrib.utils import tempdir
+
+
+def zephyr_boards() -> dict:
+    """Returns a dict mapping board to target model"""

Review Comment:
   removed get_boards



##########
python/tvm/micro/testing/pytest_plugin.py:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=invalid-name,redefined-outer-name
+""" microTVM testing fixtures used to deduce testing argument
+    values from testing parameters """
+
+import pathlib
+import json
+import os
+import datetime
+import pytest
+
+import tvm
+from tvm.contrib.utils import tempdir
+
+
+def zephyr_boards() -> dict:

Review Comment:
   that was left over, thanks for catching!



##########
tests/micro/arduino/conftest.py:
##########
@@ -15,35 +15,19 @@
 # specific language governing permissions and limitations
 # under the License.
 
-import pytest
+pytest_plugins = [
+    "tvm.micro.testing.pytest_plugin",
+]
 
-from test_utils import ARDUINO_BOARDS
+import pytest
 
 
 def pytest_addoption(parser):
-    parser.addoption(
-        "--arduino-board",
-        nargs="+",
-        required=True,
-        choices=ARDUINO_BOARDS.keys(),
-        help="Arduino board for tests.",
-    )
     parser.addoption(
         "--arduino-cli-cmd",

Review Comment:
   I think this way is more clear, specially when used in tvmc



##########
python/tvm/micro/testing/pytest_plugin.py:
##########
@@ -0,0 +1,126 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=invalid-name,redefined-outer-name
+""" microTVM testing fixtures used to deduce testing argument
+    values from testing parameters """
+
+import pathlib
+import json
+import os
+import datetime
+import pytest
+
+import tvm
+from tvm.contrib.utils import tempdir
+
+
+def zephyr_boards() -> dict:
+    """Returns a dict mapping board to target model"""
+    with open(
+        pathlib.Path(tvm.micro.get_microtvm_template_projects("zephyr")) / "boards.json"
+    ) as f:
+        board_properties = json.load(f)
+
+    boards_model = {board: info["model"] for board, info in board_properties.items()}
+    return boards_model
+
+
+def get_boards(platform: str) -> dict:
+    """Returns a dict mapping board to target model"""
+    with open(
+        pathlib.Path(tvm.micro.get_microtvm_template_projects(platform)) / "boards.json"
+    ) as f:
+        board_properties = json.load(f)
+
+    boards_model = {board: info["model"] for board, info in board_properties.items()}
+    return boards_model
+
+
+def pytest_addoption(parser):
+    """Adds more pytest arguments"""
+    parser.addoption(
+        "--board",
+        required=True,
+        choices=list(get_boards("zephyr").keys()) + list(get_boards("arduino").keys()),
+        help="microTVM boards for tests",
+    )
+    parser.addoption(
+        "--test-build-only",
+        action="store_true",
+        help="Only run tests that don't require physical hardware.",
+    )
+    parser.addoption(
+        "--tvm-debug",
+        action="store_true",
+        default=False,
+        help="If set true, it will keep the project directory for debugging.",
+    )
+
+
+@pytest.fixture(scope="module")
+def board(request):
+    return request.config.getoption("--board")
+
+
+@pytest.fixture(scope="module")

Review Comment:
   I think that makes sense. I changed them



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org