[GitHub] [tvm] driazati opened a new issue #10297: [CI Problem] `tests/micro/zephyr/test_zephyr.py::test_autotune_conv2d` is flaky

[GitBox <gi...@apache.org>]

driazati opened a new issue #10297:
URL: https://github.com/apache/tvm/issues/10297


   See [tests/micro/zephyr/test_zephyr.py::test_autotune_conv2d](https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-10292/1/pipeline/61), pytest also segfaulted out which is weird
   
   ```
   =================================== FAILURES ===================================
   ________________________ test_autotune_conv2d[qemu_x86] ________________________
   
   temp_dir = <tvm.contrib.utils.TempDirectory object at 0x7fc1b51d47b8>
   board = 'qemu_x86', west_cmd = 'west', tvm_debug = False
   
       @tvm.testing.requires_micro
       def test_autotune_conv2d(temp_dir, board, west_cmd, tvm_debug):
           """Test AutoTune for microTVM Zephyr"""
           if board != "qemu_x86":
               pytest.xfail(f"Autotune fails on {board}.")
       
           runtime = Runtime("crt", {"system-lib": True})
           model = test_utils.ZEPHYR_BOARDS[board]
           build_config = {"debug": tvm_debug}
       
           # Create a Relay model
           data_shape = (1, 3, 16, 16)
           weight_shape = (8, 3, 5, 5)
           data = relay.var("data", relay.TensorType(data_shape, "float32"))
           weight = relay.var("weight", relay.TensorType(weight_shape, "float32"))
           y = relay.nn.conv2d(
               data,
               weight,
               padding=(2, 2),
               kernel_size=(5, 5),
               kernel_layout="OIHW",
               out_dtype="float32",
           )
           f = relay.Function([data, weight], y)
           mod = tvm.IRModule.from_expr(f)
           mod = relay.transform.InferType()(mod)
       
           data_sample = np.random.rand(data_shape[0], data_shape[1], data_shape[2], data_shape[3]).astype(
               "float32"
           )
           weight_sample = np.random.rand(
               weight_shape[0], weight_shape[1], weight_shape[2], weight_shape[3]
           ).astype("float32")
           params = {mod["main"].params[1].name_hint: weight_sample}
       
           target = tvm.target.target.micro(model)
           pass_context = tvm.transform.PassContext(opt_level=3, config={"tir.disable_vectorize": True})
           with pass_context:
               tasks = tvm.autotvm.task.extract_from_program(mod["main"], {}, target)
           assert len(tasks) > 0
       
           config_main_stack_size = None
           if test_utils.qemu_boards(board):
               config_main_stack_size = 1536
       
           project_options = {
               "zephyr_board": board,
               "west_cmd": west_cmd,
               "verbose": 1,
               "project_type": "host_driven",
           }
           if config_main_stack_size is not None:
               project_options["config_main_stack_size"] = config_main_stack_size
       
           module_loader = tvm.micro.AutoTvmModuleLoader(
               template_project_dir=test_utils.TEMPLATE_PROJECT_DIR,
               project_options=project_options,
           )
       
           timeout = 200
           builder = tvm.autotvm.LocalBuilder(
               timeout=timeout,
               n_parallel=1,
               build_kwargs={"build_option": {"tir.disable_vectorize": True}},
               do_fork=True,
               build_func=tvm.micro.autotvm_build_func,
               runtime=runtime,
           )
           runner = tvm.autotvm.LocalRunner(
               number=1, repeat=1, timeout=timeout, module_loader=module_loader
           )
       
           measure_option = tvm.autotvm.measure_option(builder=builder, runner=runner)
       
           log_path = pathlib.Path("zephyr_autotune.log")
           if log_path.exists():
               log_path.unlink()
       
           n_trial = 10
           for task in tasks:
               tuner = tvm.autotvm.tuner.GATuner(task)
               tuner.tune(
                   n_trial=n_trial,
                   measure_option=measure_option,
                   callbacks=[
                       tvm.autotvm.callback.log_to_file(str(log_path)),
                       tvm.autotvm.callback.progress_bar(n_trial, si_prefix="M"),
                   ],
                   si_prefix="M",
               )
               assert tuner.best_flops > 0
       
   >       check_tune_log(log_path)
   
   tests/micro/zephyr/test_zephyr.py:471: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   log_path = PosixPath('zephyr_autotune.log')
   
       def check_tune_log(log_path: Union[pathlib.Path, str]):
           """Read the tuning log and check each result."""
           with open(log_path, "r") as f:
               lines = f.readlines()
       
           for line in lines:
               if len(line) > 0:
                   tune_result = json.loads(line)
   >               assert tune_result["result"][0][0] < 1000000000.0
   E               AssertionError
   
   python/tvm/micro/testing.py:41: AssertionError
   =============================== warnings summary ===============================
   tests/micro/zephyr/test_zephyr_aot.py::test_tflite[qemu_x86]
     /opt/tvm-venv/lib/python3.6/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
       import imp
   
   -- Docs: https://docs.pytest.org/en/stable/warnings.html
   - generated xml file: /workspace/build/pytest-results/python-microtvm-zephyr-qemu_x86-ctypes.xml -
   =========================== short test summary info ============================
   FAILED tests/micro/zephyr/test_zephyr.py::test_autotune_conv2d[qemu_x86] - As...
   ======== 1 failed, 11 passed, 1 skipped, 1 warning in 259.32s (0:04:19) ========
   tests/scripts/setup-pytest-env.sh: line 35: 29555 Segmentation fault      (core dumped) TVM_FFI=${ffi_type} python3 -m pytest -o "junit_suite_name=${test_suite_name}-${ffi_type}" "--junit-xml=${TVM_PYTEST_RESULT_DIR}/${test_suite_name}-${ffi_type}.xml" "--junit-prefix=${ffi_type}" "$@"
   script returned exit code 139
   ```
   
   cc @mehrdadh 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org