You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "guest2180 (via GitHub)" <gi...@apache.org> on 2026/06/25 07:14:39 UTC

[I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

guest2180 opened a new issue, #19887:
URL: https://github.com/apache/tvm/issues/19887

   TensorRT BYOC offload gaps on YOLO-style graphs: operators run on CUDA fallback and/or direct TensorRT, but fail to offload cleanly through TVM BYOC
   
   ## Summary
   
   We are using **TVM Relax BYOC** with **TensorRT** as the external codegen backend.
   
   This came up while trying to run a **YOLO-style model** through TVM TensorRT BYOC. After reducing the problem, we found operator-level gaps where an op can execute in CUDA fallback and/or direct TensorRT, but does not offload cleanly through the TVM TensorRT BYOC path.
   
   We found a case where an operator:
   
   1. runs correctly with normal TVM CUDA fallback
   2. runs correctly with direct TensorRT
   3. but **fails when offloaded through TVM TensorRT BYOC**
   
   The concrete operator we can prove with a self-contained reproducer is:
   
   - `relax.permute_dims`
   - `relax.image.resize2d`
   
   In addition, while working on YOLO-style graphs, we also observed related offload/partition problems around:
   
   - `relax.strided_slice`
   - `relax.concat`
   - `relax.split`
   - many `relax.reshape` glue nodes
   
   This suggests the issue is not that:
   
   - CUDA cannot execute the operator
   - TensorRT cannot execute the operator
   
   Instead, the problem appears to be in the **TVM TensorRT BYOC offload / operator mapping path**.
   
   ## Environment
   
   - TVM: local TensorRT BYOC build under `/home/perception/tvm025`, using the current `#19810` TensorRT BYOC file set
   - TensorRT: `10.16.1.11`
   - Python: `3.11`
   - Target: `cuda`
   
   ## Expected behavior
   
   If an operator can:
   
   1. run on CUDA fallback in TVM
   2. be implemented and executed directly in TensorRT
   
   then the same operator should be offloadable through TVM TensorRT BYOC, or at least fail with a clearer explanation about missing BYOC support.
   
   ## Actual behavior
   
   For `permute_dims`:
   
   - TVM CUDA fallback works
   - direct TensorRT works
   - TVM TensorRT BYOC fails with:
   
   ```text
   InternalError: Check failed: (it != map.end()) is false: tensorrt.permute_dims: Unsupported operator
   ```
   
   This means the operator reaches the TensorRT BYOC path, but the TensorRT runtime/builder side cannot handle the generated BYOC operator name.
   
   For `resize2d`:
   
   - in our current single-op BYOC reproducer, it forms a TensorRT region
   - but execution then fails with:
   
   ```text
   InternalError: Check failed: (it != map.end()) is false: tensorrt.image.resize2d: Unsupported operator
   ```
   
   - so `resize2d` has the same class of BYOC lower/runtime-side support gap as `permute_dims`
   
   For other operators seen in YOLO-style graphs, the observed problems are:
   
   1. `permute_dims`
      - explicit BYOC runtime failure
      - reaches TensorRT BYOC region
      - fails with `tensorrt.permute_dims: Unsupported operator`
   
   2. `resize2d`
      - single-op test produces `trt_regions: 1`
      - but runtime fails with `tensorrt.image.resize2d: Unsupported operator`
   
   3. `strided_slice`
      - does not form a TensorRT region in our single-op offload test
      - so it currently appears to stay on the TVM side instead of being offloaded
   
   4. `concat`
      - can be offloaded as a small standalone region in some experiments
      - but becomes problematic when trying to form larger TensorRT regions on YOLO-style graphs
   
   5. `split`
      - similarly acts as a partition boundary in YOLO-style graphs
      - it prevents larger continuous TensorRT regions from forming cleanly
   
   6. `reshape`
      - not necessarily individually unsupported
      - but YOLO-style graphs contain many reshape glue nodes, and they are part of the fragmentation problem when trying to build larger TensorRT regions
   
   ## Minimal reproduction
   
   The attached script is self-contained and demonstrates current BYOC behavior for single operators, including:
   
   1. `permute_dims`
   2. `strided_slice`
   3. `resize2d`
   
   Script:
   ```python
   from __future__ import annotations
   
   import traceback
   from dataclasses import dataclass
   
   import numpy as np
   import tvm
   from tvm import relax
   from tvm.relax import transform
   from tvm.relax.dpl import is_op, wildcard
   
   
   @dataclass(slots=True)
   class Case:
       name: str
       pattern_name: str
       mod: tvm.IRModule
       input_shape: tuple[int, ...]
   
   
   def build_permute_dims_case() -> Case:
       bb = relax.BlockBuilder()
       data = relax.Var("data", relax.TensorStructInfo((1, 3, 4, 5), "float32"))
       with bb.function("main", [data]):
           with bb.dataflow():
               out = bb.emit(relax.op.permute_dims(data, axes=[0, 2, 3, 1]))
               gv = bb.emit_output(out)
           bb.emit_func_output(gv)
       return Case(
           name="permute_dims",
           pattern_name="tensorrt.permute_dims",
           mod=bb.finalize(),
           input_shape=(1, 3, 4, 5),
       )
   
   
   def build_strided_slice_case() -> Case:
       bb = relax.BlockBuilder()
       data = relax.Var("data", relax.TensorStructInfo((1, 3, 8, 8), "float32"))
       with bb.function("main", [data]):
           with bb.dataflow():
               out = bb.emit(
                   relax.op.strided_slice(
                       data,
                       axes=[2, 3],
                       begin=[1, 1],
                       end=[7, 7],
                       strides=[2, 2],
                   )
               )
               gv = bb.emit_output(out)
           bb.emit_func_output(gv)
       return Case(
           name="strided_slice",
           pattern_name="tensorrt.strided_slice",
           mod=bb.finalize(),
           input_shape=(1, 3, 8, 8),
       )
   
   
   def build_resize2d_case() -> Case:
       bb = relax.BlockBuilder()
       data = relax.Var("data", relax.TensorStructInfo((1, 3, 4, 4), "float32"))
       with bb.function("main", [data]):
           with bb.dataflow():
               out = bb.emit(
                   relax.op.image.resize2d(
                       data,
                       size=(8, 8),
                       layout="NCHW",
                       method="nearest_neighbor",
                       coordinate_transformation_mode="asymmetric",
                       rounding_method="floor",
                   )
               )
               gv = bb.emit_output(out)
           bb.emit_func_output(gv)
       return Case(
           name="resize2d",
           pattern_name="tensorrt.image.resize2d",
           mod=bb.finalize(),
           input_shape=(1, 3, 4, 4),
       )
   
   
   def build_concat_case() -> Case:
       bb = relax.BlockBuilder()
       data = relax.Var("data", relax.TensorStructInfo((1, 3, 4, 4), "float32"))
       with bb.function("main", [data]):
           with bb.dataflow():
               out = bb.emit(relax.op.concat((data, data), axis=1))
               gv = bb.emit_output(out)
           bb.emit_func_output(gv)
       return Case(
           name="concat",
           pattern_name="tensorrt.concatenate",
           mod=bb.finalize(),
           input_shape=(1, 3, 4, 4),
       )
   
   
   def partition_single_op(case: Case) -> tvm.IRModule:
       if case.name == "concat":
           pattern = is_op("relax.concat")(wildcard())
       else:
           pattern = is_op(f"relax.{case.name}")(wildcard()) if case.name != "resize2d" else is_op("relax.image.resize2d")(wildcard(), wildcard())
       # Build patterns explicitly per case to avoid relying on local helper files.
       if case.name == "permute_dims":
           patterns = [(case.pattern_name, is_op("relax.permute_dims")(wildcard()))]
       elif case.name == "strided_slice":
           patterns = [(case.pattern_name, is_op("relax.strided_slice")(wildcard()))]
       elif case.name == "resize2d":
           patterns = [(case.pattern_name, is_op("relax.image.resize2d")(wildcard(), wildcard()))]
       elif case.name == "concat":
           patterns = [(case.pattern_name, is_op("relax.concat")(wildcard()))]
       else:
           raise ValueError(case.name)
   
       mod = transform.FuseOpsByPattern(
           patterns,
           bind_constants=True,
           annotate_codegen=False,
       )(case.mod)
       mod = transform.MergeCompositeFunctions()(mod)
       mod = transform.RunCodegen()(mod)
       return mod
   
   
   def count_trt_regions(mod: tvm.IRModule) -> int:
       return mod.script().count('R.call_dps_packed("')
   
   
   def run_case(case: Case) -> None:
       print(f"\n=== case: {case.name} ===")
       try:
           lowered = partition_single_op(case)
           print("run_codegen_ok: true")
           print("trt_regions:", count_trt_regions(lowered))
   
           with tvm.transform.PassContext(opt_level=0):
               ex = tvm.compile(lowered, "cuda")
           dev = tvm.cuda(0)
           vm = relax.VirtualMachine(ex, dev)
           inp = np.random.rand(*case.input_shape).astype("float32")
           out = vm["main"](tvm.runtime.tensor(inp, dev))
           print("runtime_ok: true")
           if hasattr(out, "shape"):
               print("output_shape:", out.shape)
           else:
               print("output_type:", type(out).__name__)
       except Exception as err:  # noqa: BLE001
           print("runtime_ok: false")
           print("error_type:", type(err).__name__)
           print("error_message:", err)
           print("[traceback_begin]")
           print(traceback.format_exc().rstrip())
           print("[traceback_end]")
   
   
   def main() -> None:
       cases = [
           build_permute_dims_case(),
           build_strided_slice_case(),
           build_resize2d_case(),
       ]
       for case in cases:
           run_case(case)
   
   
   if __name__ == "__main__":
       main()
   ```
   
   ## Repro command
   
   ```bash
   export TVM_HOME=/path/to/tvm
   export PYTHONPATH=$TVM_HOME/python:$PYTHONPATH
   
   python repro_trt_single_op_offload.py
   ```
   
   ## Observed output
   
   ```text
   === case: permute_dims ===
   run_codegen_ok: true
   trt_regions: 1
   runtime_ok: false
   error_type: InternalError
   error_message: Check failed: (it != map.end()) is false: tensorrt.permute_dims: Unsupported operator
   
   === case: resize2d ===
   run_codegen_ok: true
   trt_regions: 1
   runtime_ok: false
   error_type: InternalError
   error_message: Check failed: (it != map.end()) is false: tensorrt.image.resize2d: Unsupported operator
   ```
   
   ## Why this demonstrates a BYOC offload gap
   
   These reproducers isolate two distinct offload-gap modes:
   
   - `permute_dims`: the op reaches the TensorRT BYOC runtime path, but fails there as unsupported
   - `resize2d`: the op also reaches the TensorRT BYOC runtime path, but fails there as unsupported
   - `strided_slice`: the op does not form a TensorRT region and therefore stays on the TVM side
   
   So this is not just a generic runtime limitation of CUDA or TensorRT.
   
   It is specifically a **TVM TensorRT BYOC offload support gap** in the partitioning / lowering path.
   
   ## Additional YOLO context
   
   This issue was found while trying to run a YOLO-style model with TensorRT BYOC.
   
   On that graph, the main difficulty is not the backbone convolution operators, but the neck/tail glue operators. In practice, these operators fragment the partition or fail to lower cleanly when we try to build larger TensorRT regions.
   
   Examples observed on the YOLO-style graph include:
   
   - `permute_dims`
   - `image.resize2d`
   - `strided_slice`
   - `concat`
   - `split`
   - many `reshape`
   
   So while the minimal reproducer here uses `permute_dims`, the broader real-world motivation is that these gaps make YOLO-style TensorRT BYOC offload much harder than expected.
   
   ## Notes
   
   One of the failures is currently triggered during the TensorRT BYOC execution path with:
   
   ```text
   tensorrt.permute_dims: Unsupported operator
   ```
   
   This may indicate a mismatch between:
   
   1. the BYOC pattern/operator name emitted by Relax partitioning
   2. the operator names registered in the TensorRT runtime converter table
   
   If helpful, I can also provide similar minimal reproducers for other YOLO-related operators, especially `concat` and `split`, or the full model and script I used to test yolo.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4828975079

   @tlopex 
   ### Repro model
   
   Official Ultralytics model:
   
   - `https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.onnx`
   
   ### Repro script
   
   ```python
   from __future__ import annotations
   
   import argparse
   import os
   import traceback
   from collections import Counter
   from pathlib import Path
   
   import onnx
   import tvm
   from tvm import relax
   from tvm.relax.dpl import is_op, wildcard
   from tvm.relax.frontend.onnx import from_onnx
   
   
   TRT_LIB_DIR = "/usr/local/TensorRT-10.16.1.11/lib"
   DEFAULT_ONNX = Path("/home/perception/yolo11n_repro/yolo11n.onnx")
   INPUT_NAME = "images"
   INPUT_SHAPE = [1, 3, 640, 640]
   
   
   def ensure_tensorrt_library_path() -> None:
       if not os.path.isdir(TRT_LIB_DIR):
           return
       current = os.environ.get("LD_LIBRARY_PATH", "")
       parts = [p for p in current.split(":") if p]
       if TRT_LIB_DIR not in parts:
           os.environ["LD_LIBRARY_PATH"] = f"{TRT_LIB_DIR}:{current}" if current else TRT_LIB_DIR
   
   
   def inspect_onnx_model(onnx_path: Path) -> dict[str, object]:
       model = onnx.load(str(onnx_path))
       return {
           "model": model,
           "inputs": [
               (value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
               for value in model.graph.input
           ],
           "outputs": [
               (value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
               for value in model.graph.output
           ],
           "node_count": len(model.graph.node),
           "opset": [(item.domain or "ai.onnx", item.version) for item in model.opset_import],
           "ops": Counter(node.op_type for node in model.graph.node),
       }
   
   
   def build_base_patterns() -> list[tuple[str, object]]:
       return [
           ("tensorrt.nn.conv2d", is_op("relax.nn.conv2d")(wildcard(), wildcard())),
           ("tensorrt.nn.conv2d_transpose", is_op("relax.nn.conv2d_transpose")(wildcard(), wildcard())),
           ("tensorrt.add", is_op("relax.add")(wildcard(), wildcard())),
           ("tensorrt.subtract", is_op("relax.subtract")(wildcard(), wildcard())),
           ("tensorrt.multiply", is_op("relax.multiply")(wildcard(), wildcard())),
           ("tensorrt.divide", is_op("relax.divide")(wildcard(), wildcard())),
           ("tensorrt.sigmoid", is_op("relax.sigmoid")(wildcard())),
           ("tensorrt.nn.softmax", is_op("relax.nn.softmax")(wildcard())),
           ("tensorrt.nn.max_pool2d", is_op("relax.nn.max_pool2d")(wildcard())),
           ("tensorrt.nn.avg_pool2d", is_op("relax.nn.avg_pool2d")(wildcard())),
       ]
   
   
   def build_extra_pattern(name: str) -> tuple[str, object]:
       mapping = {
           "relu": ("tensorrt.nn.relu", is_op("relax.nn.relu")(wildcard())),
           "concat": ("tensorrt.concatenate", is_op("relax.concat")(wildcard())),
           "split": ("tensorrt.split", is_op("relax.split")(wildcard())),
           "exp": ("tensorrt.exp", is_op("relax.exp")(wildcard())),
           "atan": ("tensorrt.atan", is_op("relax.atan")(wildcard())),
           "resize2d": ("tensorrt.image.resize2d", is_op("relax.image.resize2d")(wildcard())),
           "permute_dims": ("tensorrt.permute_dims", is_op("relax.permute_dims")(wildcard())),
           "reshape": ("tensorrt.reshape", is_op("relax.reshape")(wildcard())),
           "expand_dims": ("tensorrt.expand_dims", is_op("relax.expand_dims")(wildcard())),
       }
       return mapping[name]
   
   
   def count_trt_regions(mod: tvm.IRModule) -> int:
       count = 0
       for _, func in mod.functions.items():
           attrs = getattr(func, "attrs", None)
           if attrs is None:
               continue
           try:
               if attrs["Codegen"] == "tensorrt":
                   count += 1
           except Exception:
               pass
       return count
   
   
   def try_patterns(onnx_path: Path, extra_names: list[str]) -> dict[str, object]:
       info = inspect_onnx_model(onnx_path)
       patterns = build_base_patterns() + [build_extra_pattern(name) for name in extra_names]
       result: dict[str, object] = {"extras": list(extra_names)}
       try:
           mod = from_onnx(info["model"], shape_dict={INPUT_NAME: INPUT_SHAPE})
           fused = relax.transform.FuseOpsByPattern(patterns)(mod)
           merged = relax.transform.MergeCompositeFunctions()(fused)
           result["trt_regions"] = count_trt_regions(merged)
           _ = relax.transform.RunCodegen()(merged)
           result["status"] = "ok"
       except Exception as err:
           result["status"] = "failed"
           result["error"] = str(err)
           result["trace"] = traceback.format_exc()
       return result
   
   
   def print_model_summary(onnx_path: Path) -> None:
       info = inspect_onnx_model(onnx_path)
       print(f"onnx={onnx_path}")
       print(f"inputs={info['inputs']}")
       print(f"outputs={info['outputs']}")
       print(f"nodes={info['node_count']}")
       print(f"opset={info['opset']}")
       print(f"top_ops={info['ops'].most_common(12)}")
   
   
   def run_baseline(onnx_path: Path) -> int:
       print("mode=baseline")
       result = try_patterns(onnx_path, [])
       if result["status"] == "ok":
           print(f"baseline=OK trt_regions={result['trt_regions']}")
           return 0
       print(f"baseline=FAIL trt_regions={result.get('trt_regions', 'n/a')}")
       print(f"baseline_error={result['error']}")
       return 1
   
   
   def run_pattern_scan(onnx_path: Path) -> int:
       print("mode=pattern_scan")
       scan_items = [
           "concat",
           "split",
           "resize2d",
           "permute_dims",
           "reshape",
           "expand_dims",
           "relu",
           "exp",
           "atan",
       ]
       serializer_hits = 0
       for name in scan_items:
           result = try_patterns(onnx_path, [name])
           if result["status"] == "ok":
               print(f"scan[{name}]=OK trt_regions={result['trt_regions']}")
               continue
           error = str(result.get("error", ""))
           if "Cannot find the name of the constant" in error:
               serializer_hits += 1
               kind = "SERIALIZER_CONST_NAME"
           else:
               kind = "OTHER_FAIL"
           print(f"scan[{name}]={kind} trt_regions={result.get('trt_regions', 'n/a')}")
           print(f"scan_error[{name}]={error}")
       print(f"serializer_const_name_hits={serializer_hits}")
       return 0
   
   
   def main() -> int:
       parser = argparse.ArgumentParser(description="Minimal YOLO11n TensorRT BYOC serializer repro")
       parser.add_argument("--onnx", type=Path, default=DEFAULT_ONNX)
       parser.add_argument("--mode", choices=["baseline", "pattern_scan"], default="pattern_scan")
       args = parser.parse_args()
   
       ensure_tensorrt_library_path()
       print_model_summary(args.onnx)
   
       if args.mode == "baseline":
           return run_baseline(args.onnx)
       return run_pattern_scan(args.onnx)
   
   
   if __name__ == "__main__":
       raise SystemExit(main())
   
   ```
   
   ### Repro command
   
   ```bash
   python repro_yolo11n_byoc.py --mode pattern_scan
   ```
   `--mode baseline` runs the same graph with only the stable base TensorRT patterns, as a control case.
   
   ### Observed result
   
   ```text
   scan[concat]=SERIALIZER_CONST_NAME trt_regions=21
   scan_error[concat]=Check failed: (name != constant_names_.end()) is false: Cannot find the name of the constant: metadata["relax.expr.Constant"][0]
   
   scan[split]=SERIALIZER_CONST_NAME trt_regions=44
   scan_error[split]=Check failed: (name != constant_names_.end()) is false: Cannot find the name of the constant: metadata["relax.expr.Constant"][0]
   
   scan[resize2d]=OK trt_regions=42
   scan[permute_dims]=OK trt_regions=44
   ```
   
   ### Interpretation
   
   - `concat` / `split` reproduce a serializer constant-name failure
   - `resize2d` / `permute_dims` do not reproduce that same failure on this model
   - In our smaller handcrafted repros, `resize2d` / `permute_dims` still hit unsupported offload/codegen paths
   
   So these currently look like two different TensorRT BYOC issues, not one.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4967254249

   @tlopex I tested yolo11n-seg with the script used in the previous reply and found that the existing tuple-output remap could change the semantics of existing TupleGetItem indices, so I updated the fix accordingly. The unit tests and several small models now run successfully. I am attaching the patch in this comment because submitting it to upstream TVM or to your private fork is too annoying.
   
   [relax_byoc_tuple_output_fixes.patch](https://github.com/user-attachments/files/30000714/relax_byoc_tuple_output_fixes.patch)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4972720122

   Thanks! @guest2180 I just updated according to what you said, could you try again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4802440467

   Thanks for reporting, will take a look afterwards


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4962940233

   I just send a pr #19998 to fix , could you have a look and try whether it works?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4824480817

   @guest2180  To address the YOLO case in one go rather than op-by-op, could you share the complete list of operators in your model or you think can be used commonly that don't offload to TensorRT? Since I found that we have to write convert for every op for tensortrt that be used, I am not sure that if it's just like this or it's the usual pattern, please correct me if I'm wrong . If you are interested, you are also welcome to discuss with me or contribute.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4978365695

   @tlopex  I tested the new patch, and it looks excellent. YOLO-style subgraphs can now be fully offloaded, and the initial latency numbers are comparable to native TensorRT.
   I then tried a point-cloud model and found that it is still some distance away from full-graph offload, so I prepared these two additional patches. They include new operator support, onnx changes, runtime changes, and added unit tests. Please feel free to evaluate the risks and merge the changes if they look reasonable. I also attached the script I used to run the point-cloud model.
   I do not have other suitable models on hand for further testing right now, so I do not have more feedback beyond this.
   
   [byoc_relax_correctness_full_offload.patch](https://github.com/user-attachments/files/30040189/byoc_relax_correctness_full_offload.patch)
   
   [tensorrt_runtime_stability_cache.patch](https://github.com/user-attachments/files/30040184/tensorrt_runtime_stability_cache.patch)
   
   [test_pointpillars.py](https://github.com/user-attachments/files/30040194/test_pointpillars.py)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4923343891

   @tlopex 
   Update from our side after testing a newer local TVM checkout around commit `ad87f9b`:
   
   The unsupported-op / naming mismatch part appears to be improved. In particular, the earlier TensorRT BYOC failures around `permute_dims` / `image.resize2d` reaching BYOC but then failing as unsupported are no longer the first blocking issue in our setup.
   
   However, when we switch to the official Relax TensorRT partition flow on a YOLO11 segmentation model, we now hit a different blocker earlier in the BYOC pipeline:
   
   Model:
   https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n-seg.onnx
   
   Test script:
   ```
   from __future__ import annotations
   
   import argparse
   import traceback
   from collections import Counter
   from pathlib import Path
   
   import onnx
   from tvm.relax.backend.contrib.tensorrt import partition_for_tensorrt
   from tvm.relax.frontend.onnx import from_onnx
   
   
   DEFAULT_ONNX = Path("/home/perception/yolo11n_repro/yolo11n-seg.onnx")
   DEFAULT_INPUT_NAME = "images"
   DEFAULT_INPUT_SHAPE = [1, 3, 640, 640]
   
   
   def inspect_onnx_model(onnx_path: Path) -> dict[str, object]:
       model = onnx.load(str(onnx_path))
       return {
           "model": model,
           "inputs": [
               (value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
               for value in model.graph.input
           ],
           "outputs": [
               (value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
               for value in model.graph.output
           ],
           "node_count": len(model.graph.node),
           "opset": [(item.domain or "ai.onnx", item.version) for item in model.opset_import],
           "ops": Counter(node.op_type for node in model.graph.node),
       }
   
   
   def parse_shape(text: str) -> list[int]:
       return [int(part.strip()) for part in text.split(",") if part.strip()]
   
   
   def main() -> int:
       parser = argparse.ArgumentParser(
           description="Reproduce the official Relax TensorRT partition cycle on yolo11n-seg.onnx"
       )
       parser.add_argument("--onnx", type=Path, default=DEFAULT_ONNX)
       parser.add_argument("--input-name", default=DEFAULT_INPUT_NAME)
       parser.add_argument("--input-shape", default="1,3,640,640")
       parser.add_argument("--show-trace", action="store_true")
       args = parser.parse_args()
   
       input_shape = parse_shape(args.input_shape)
       info = inspect_onnx_model(args.onnx)
   
       print(f"onnx={args.onnx}")
       print(f"inputs={info['inputs']}")
       print(f"outputs={info['outputs']}")
       print(f"nodes={info['node_count']}")
       print(f"opset={info['opset']}")
       print(f"top_ops={info['ops'].most_common(12)}")
       print("step=from_onnx")
   
       mod = from_onnx(info["model"], shape_dict={args.input_name: input_shape})
       print("from_onnx=ok")
       print("step=partition_for_tensorrt")
   
       try:
           _ = partition_for_tensorrt(mod)
       except Exception as err:
           print(f"partition=FAIL type={type(err).__name__}")
           print(f"error={err}")
           if args.show_trace:
               print("traceback_begin")
               print(traceback.format_exc())
               print("traceback_end")
           return 1
   
       print("partition=OK")
       return 0
   
   
   if __name__ == "__main__":
       raise SystemExit(main())
   ```
   python repro_yolo11seg_native_cuda.py --opt-level 0 --show-trace
   
   What we see:
   1. `from_onnx(...)` succeeds
   2. native TVM CUDA compilation/execution succeeds
   3. but `tvm.relax.backend.contrib.tensorrt.partition_for_tensorrt(mod)` fails in `MergeCompositeFunctions()` with a cyclic-dependency error
   
   Error:
   `InternalError: Check failed: (depgroup != cur_group) is false: A cyclic dependency detected between the groups lv13 and lv77 are in.`
   
   So from our current testing, this looks like progress: the BYOC operator-name / runtime-converter mismatch is no longer the main blocker, but larger YOLO-style graphs now run into a Relax partition/merge issue (likely around `concat` / `split`-heavy subgraph formation) before codegen.
   
   Separately, the plain detection model also exposes another front-end issue:
   
   https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.onnx
   
   This one currently does not get past `from_onnx(...)` in our environment. It fails while converting `Range` into Relax `arange`, with a type mismatch on the dtype argument (`Expected DataType but got ir.PrimType`).
   
   So at this point we are seeing three stages of progress / remaining issues:
   - the original BYOC unsupported-op naming problem is improved
   - `yolo11n-seg.onnx` now reaches official TensorRT partitioning but fails with a cyclic dependency during `MergeCompositeFunctions`
   - `yolo11n.onnx` still fails earlier in the ONNX importer on `Range`
   
   One additional note from our side: for relatively simple unsupported ops such as `resize2d` or some small shape/dim-related glue handling, it is often still practical for downstream users to do small model-specific or project-specific hand-written workarounds. Those cases may not always require a full official TVM-side update immediately.
   
   The cyclic-dependency problem feels different. Once the graph reaches `MergeCompositeFunctions()` and fails with a group-dependency cycle on official partitioning, that is beyond what we can reasonably patch around at the model/project level. It likely needs a more formal fix in the Relax partition / merge pipeline itself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "MasterJH5574 (via GitHub)" <gi...@apache.org>.
MasterJH5574 closed issue #19887: [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning
URL: https://github.com/apache/tvm/issues/19887


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4975708193

   @tlopex   I tried and everything seems ok, btw the current merge rules may split otherwise-supported regions to avoid cyclic dependencies, so even if all operators are supported by the BYOC backend, the whole model may still not be offloaded as a single subgraph. I’m not sure how much this affects steady-state inference latency, but it can increase the initial engine build time because multiple engines/subgraphs need to be built.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4976911374

   Just updated again, you can see and have a try, and give some feedbacks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org