You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "guest2180 (via GitHub)" <gi...@apache.org> on 2026/06/25 07:14:39 UTC
[I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
guest2180 opened a new issue, #19887:
URL: https://github.com/apache/tvm/issues/19887
TensorRT BYOC offload gaps on YOLO-style graphs: operators run on CUDA fallback and/or direct TensorRT, but fail to offload cleanly through TVM BYOC
## Summary
We are using **TVM Relax BYOC** with **TensorRT** as the external codegen backend.
This came up while trying to run a **YOLO-style model** through TVM TensorRT BYOC. After reducing the problem, we found operator-level gaps where an op can execute in CUDA fallback and/or direct TensorRT, but does not offload cleanly through the TVM TensorRT BYOC path.
We found a case where an operator:
1. runs correctly with normal TVM CUDA fallback
2. runs correctly with direct TensorRT
3. but **fails when offloaded through TVM TensorRT BYOC**
The concrete operator we can prove with a self-contained reproducer is:
- `relax.permute_dims`
- `relax.image.resize2d`
In addition, while working on YOLO-style graphs, we also observed related offload/partition problems around:
- `relax.strided_slice`
- `relax.concat`
- `relax.split`
- many `relax.reshape` glue nodes
This suggests the issue is not that:
- CUDA cannot execute the operator
- TensorRT cannot execute the operator
Instead, the problem appears to be in the **TVM TensorRT BYOC offload / operator mapping path**.
## Environment
- TVM: local TensorRT BYOC build under `/home/perception/tvm025`, using the current `#19810` TensorRT BYOC file set
- TensorRT: `10.16.1.11`
- Python: `3.11`
- Target: `cuda`
## Expected behavior
If an operator can:
1. run on CUDA fallback in TVM
2. be implemented and executed directly in TensorRT
then the same operator should be offloadable through TVM TensorRT BYOC, or at least fail with a clearer explanation about missing BYOC support.
## Actual behavior
For `permute_dims`:
- TVM CUDA fallback works
- direct TensorRT works
- TVM TensorRT BYOC fails with:
```text
InternalError: Check failed: (it != map.end()) is false: tensorrt.permute_dims: Unsupported operator
```
This means the operator reaches the TensorRT BYOC path, but the TensorRT runtime/builder side cannot handle the generated BYOC operator name.
For `resize2d`:
- in our current single-op BYOC reproducer, it forms a TensorRT region
- but execution then fails with:
```text
InternalError: Check failed: (it != map.end()) is false: tensorrt.image.resize2d: Unsupported operator
```
- so `resize2d` has the same class of BYOC lower/runtime-side support gap as `permute_dims`
For other operators seen in YOLO-style graphs, the observed problems are:
1. `permute_dims`
- explicit BYOC runtime failure
- reaches TensorRT BYOC region
- fails with `tensorrt.permute_dims: Unsupported operator`
2. `resize2d`
- single-op test produces `trt_regions: 1`
- but runtime fails with `tensorrt.image.resize2d: Unsupported operator`
3. `strided_slice`
- does not form a TensorRT region in our single-op offload test
- so it currently appears to stay on the TVM side instead of being offloaded
4. `concat`
- can be offloaded as a small standalone region in some experiments
- but becomes problematic when trying to form larger TensorRT regions on YOLO-style graphs
5. `split`
- similarly acts as a partition boundary in YOLO-style graphs
- it prevents larger continuous TensorRT regions from forming cleanly
6. `reshape`
- not necessarily individually unsupported
- but YOLO-style graphs contain many reshape glue nodes, and they are part of the fragmentation problem when trying to build larger TensorRT regions
## Minimal reproduction
The attached script is self-contained and demonstrates current BYOC behavior for single operators, including:
1. `permute_dims`
2. `strided_slice`
3. `resize2d`
Script:
```python
from __future__ import annotations
import traceback
from dataclasses import dataclass
import numpy as np
import tvm
from tvm import relax
from tvm.relax import transform
from tvm.relax.dpl import is_op, wildcard
@dataclass(slots=True)
class Case:
name: str
pattern_name: str
mod: tvm.IRModule
input_shape: tuple[int, ...]
def build_permute_dims_case() -> Case:
bb = relax.BlockBuilder()
data = relax.Var("data", relax.TensorStructInfo((1, 3, 4, 5), "float32"))
with bb.function("main", [data]):
with bb.dataflow():
out = bb.emit(relax.op.permute_dims(data, axes=[0, 2, 3, 1]))
gv = bb.emit_output(out)
bb.emit_func_output(gv)
return Case(
name="permute_dims",
pattern_name="tensorrt.permute_dims",
mod=bb.finalize(),
input_shape=(1, 3, 4, 5),
)
def build_strided_slice_case() -> Case:
bb = relax.BlockBuilder()
data = relax.Var("data", relax.TensorStructInfo((1, 3, 8, 8), "float32"))
with bb.function("main", [data]):
with bb.dataflow():
out = bb.emit(
relax.op.strided_slice(
data,
axes=[2, 3],
begin=[1, 1],
end=[7, 7],
strides=[2, 2],
)
)
gv = bb.emit_output(out)
bb.emit_func_output(gv)
return Case(
name="strided_slice",
pattern_name="tensorrt.strided_slice",
mod=bb.finalize(),
input_shape=(1, 3, 8, 8),
)
def build_resize2d_case() -> Case:
bb = relax.BlockBuilder()
data = relax.Var("data", relax.TensorStructInfo((1, 3, 4, 4), "float32"))
with bb.function("main", [data]):
with bb.dataflow():
out = bb.emit(
relax.op.image.resize2d(
data,
size=(8, 8),
layout="NCHW",
method="nearest_neighbor",
coordinate_transformation_mode="asymmetric",
rounding_method="floor",
)
)
gv = bb.emit_output(out)
bb.emit_func_output(gv)
return Case(
name="resize2d",
pattern_name="tensorrt.image.resize2d",
mod=bb.finalize(),
input_shape=(1, 3, 4, 4),
)
def build_concat_case() -> Case:
bb = relax.BlockBuilder()
data = relax.Var("data", relax.TensorStructInfo((1, 3, 4, 4), "float32"))
with bb.function("main", [data]):
with bb.dataflow():
out = bb.emit(relax.op.concat((data, data), axis=1))
gv = bb.emit_output(out)
bb.emit_func_output(gv)
return Case(
name="concat",
pattern_name="tensorrt.concatenate",
mod=bb.finalize(),
input_shape=(1, 3, 4, 4),
)
def partition_single_op(case: Case) -> tvm.IRModule:
if case.name == "concat":
pattern = is_op("relax.concat")(wildcard())
else:
pattern = is_op(f"relax.{case.name}")(wildcard()) if case.name != "resize2d" else is_op("relax.image.resize2d")(wildcard(), wildcard())
# Build patterns explicitly per case to avoid relying on local helper files.
if case.name == "permute_dims":
patterns = [(case.pattern_name, is_op("relax.permute_dims")(wildcard()))]
elif case.name == "strided_slice":
patterns = [(case.pattern_name, is_op("relax.strided_slice")(wildcard()))]
elif case.name == "resize2d":
patterns = [(case.pattern_name, is_op("relax.image.resize2d")(wildcard(), wildcard()))]
elif case.name == "concat":
patterns = [(case.pattern_name, is_op("relax.concat")(wildcard()))]
else:
raise ValueError(case.name)
mod = transform.FuseOpsByPattern(
patterns,
bind_constants=True,
annotate_codegen=False,
)(case.mod)
mod = transform.MergeCompositeFunctions()(mod)
mod = transform.RunCodegen()(mod)
return mod
def count_trt_regions(mod: tvm.IRModule) -> int:
return mod.script().count('R.call_dps_packed("')
def run_case(case: Case) -> None:
print(f"\n=== case: {case.name} ===")
try:
lowered = partition_single_op(case)
print("run_codegen_ok: true")
print("trt_regions:", count_trt_regions(lowered))
with tvm.transform.PassContext(opt_level=0):
ex = tvm.compile(lowered, "cuda")
dev = tvm.cuda(0)
vm = relax.VirtualMachine(ex, dev)
inp = np.random.rand(*case.input_shape).astype("float32")
out = vm["main"](tvm.runtime.tensor(inp, dev))
print("runtime_ok: true")
if hasattr(out, "shape"):
print("output_shape:", out.shape)
else:
print("output_type:", type(out).__name__)
except Exception as err: # noqa: BLE001
print("runtime_ok: false")
print("error_type:", type(err).__name__)
print("error_message:", err)
print("[traceback_begin]")
print(traceback.format_exc().rstrip())
print("[traceback_end]")
def main() -> None:
cases = [
build_permute_dims_case(),
build_strided_slice_case(),
build_resize2d_case(),
]
for case in cases:
run_case(case)
if __name__ == "__main__":
main()
```
## Repro command
```bash
export TVM_HOME=/path/to/tvm
export PYTHONPATH=$TVM_HOME/python:$PYTHONPATH
python repro_trt_single_op_offload.py
```
## Observed output
```text
=== case: permute_dims ===
run_codegen_ok: true
trt_regions: 1
runtime_ok: false
error_type: InternalError
error_message: Check failed: (it != map.end()) is false: tensorrt.permute_dims: Unsupported operator
=== case: resize2d ===
run_codegen_ok: true
trt_regions: 1
runtime_ok: false
error_type: InternalError
error_message: Check failed: (it != map.end()) is false: tensorrt.image.resize2d: Unsupported operator
```
## Why this demonstrates a BYOC offload gap
These reproducers isolate two distinct offload-gap modes:
- `permute_dims`: the op reaches the TensorRT BYOC runtime path, but fails there as unsupported
- `resize2d`: the op also reaches the TensorRT BYOC runtime path, but fails there as unsupported
- `strided_slice`: the op does not form a TensorRT region and therefore stays on the TVM side
So this is not just a generic runtime limitation of CUDA or TensorRT.
It is specifically a **TVM TensorRT BYOC offload support gap** in the partitioning / lowering path.
## Additional YOLO context
This issue was found while trying to run a YOLO-style model with TensorRT BYOC.
On that graph, the main difficulty is not the backbone convolution operators, but the neck/tail glue operators. In practice, these operators fragment the partition or fail to lower cleanly when we try to build larger TensorRT regions.
Examples observed on the YOLO-style graph include:
- `permute_dims`
- `image.resize2d`
- `strided_slice`
- `concat`
- `split`
- many `reshape`
So while the minimal reproducer here uses `permute_dims`, the broader real-world motivation is that these gaps make YOLO-style TensorRT BYOC offload much harder than expected.
## Notes
One of the failures is currently triggered during the TensorRT BYOC execution path with:
```text
tensorrt.permute_dims: Unsupported operator
```
This may indicate a mismatch between:
1. the BYOC pattern/operator name emitted by Relax partitioning
2. the operator names registered in the TensorRT runtime converter table
If helpful, I can also provide similar minimal reproducers for other YOLO-related operators, especially `concat` and `split`, or the full model and script I used to test yolo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4828975079
@tlopex
### Repro model
Official Ultralytics model:
- `https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.onnx`
### Repro script
```python
from __future__ import annotations
import argparse
import os
import traceback
from collections import Counter
from pathlib import Path
import onnx
import tvm
from tvm import relax
from tvm.relax.dpl import is_op, wildcard
from tvm.relax.frontend.onnx import from_onnx
TRT_LIB_DIR = "/usr/local/TensorRT-10.16.1.11/lib"
DEFAULT_ONNX = Path("/home/perception/yolo11n_repro/yolo11n.onnx")
INPUT_NAME = "images"
INPUT_SHAPE = [1, 3, 640, 640]
def ensure_tensorrt_library_path() -> None:
if not os.path.isdir(TRT_LIB_DIR):
return
current = os.environ.get("LD_LIBRARY_PATH", "")
parts = [p for p in current.split(":") if p]
if TRT_LIB_DIR not in parts:
os.environ["LD_LIBRARY_PATH"] = f"{TRT_LIB_DIR}:{current}" if current else TRT_LIB_DIR
def inspect_onnx_model(onnx_path: Path) -> dict[str, object]:
model = onnx.load(str(onnx_path))
return {
"model": model,
"inputs": [
(value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
for value in model.graph.input
],
"outputs": [
(value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
for value in model.graph.output
],
"node_count": len(model.graph.node),
"opset": [(item.domain or "ai.onnx", item.version) for item in model.opset_import],
"ops": Counter(node.op_type for node in model.graph.node),
}
def build_base_patterns() -> list[tuple[str, object]]:
return [
("tensorrt.nn.conv2d", is_op("relax.nn.conv2d")(wildcard(), wildcard())),
("tensorrt.nn.conv2d_transpose", is_op("relax.nn.conv2d_transpose")(wildcard(), wildcard())),
("tensorrt.add", is_op("relax.add")(wildcard(), wildcard())),
("tensorrt.subtract", is_op("relax.subtract")(wildcard(), wildcard())),
("tensorrt.multiply", is_op("relax.multiply")(wildcard(), wildcard())),
("tensorrt.divide", is_op("relax.divide")(wildcard(), wildcard())),
("tensorrt.sigmoid", is_op("relax.sigmoid")(wildcard())),
("tensorrt.nn.softmax", is_op("relax.nn.softmax")(wildcard())),
("tensorrt.nn.max_pool2d", is_op("relax.nn.max_pool2d")(wildcard())),
("tensorrt.nn.avg_pool2d", is_op("relax.nn.avg_pool2d")(wildcard())),
]
def build_extra_pattern(name: str) -> tuple[str, object]:
mapping = {
"relu": ("tensorrt.nn.relu", is_op("relax.nn.relu")(wildcard())),
"concat": ("tensorrt.concatenate", is_op("relax.concat")(wildcard())),
"split": ("tensorrt.split", is_op("relax.split")(wildcard())),
"exp": ("tensorrt.exp", is_op("relax.exp")(wildcard())),
"atan": ("tensorrt.atan", is_op("relax.atan")(wildcard())),
"resize2d": ("tensorrt.image.resize2d", is_op("relax.image.resize2d")(wildcard())),
"permute_dims": ("tensorrt.permute_dims", is_op("relax.permute_dims")(wildcard())),
"reshape": ("tensorrt.reshape", is_op("relax.reshape")(wildcard())),
"expand_dims": ("tensorrt.expand_dims", is_op("relax.expand_dims")(wildcard())),
}
return mapping[name]
def count_trt_regions(mod: tvm.IRModule) -> int:
count = 0
for _, func in mod.functions.items():
attrs = getattr(func, "attrs", None)
if attrs is None:
continue
try:
if attrs["Codegen"] == "tensorrt":
count += 1
except Exception:
pass
return count
def try_patterns(onnx_path: Path, extra_names: list[str]) -> dict[str, object]:
info = inspect_onnx_model(onnx_path)
patterns = build_base_patterns() + [build_extra_pattern(name) for name in extra_names]
result: dict[str, object] = {"extras": list(extra_names)}
try:
mod = from_onnx(info["model"], shape_dict={INPUT_NAME: INPUT_SHAPE})
fused = relax.transform.FuseOpsByPattern(patterns)(mod)
merged = relax.transform.MergeCompositeFunctions()(fused)
result["trt_regions"] = count_trt_regions(merged)
_ = relax.transform.RunCodegen()(merged)
result["status"] = "ok"
except Exception as err:
result["status"] = "failed"
result["error"] = str(err)
result["trace"] = traceback.format_exc()
return result
def print_model_summary(onnx_path: Path) -> None:
info = inspect_onnx_model(onnx_path)
print(f"onnx={onnx_path}")
print(f"inputs={info['inputs']}")
print(f"outputs={info['outputs']}")
print(f"nodes={info['node_count']}")
print(f"opset={info['opset']}")
print(f"top_ops={info['ops'].most_common(12)}")
def run_baseline(onnx_path: Path) -> int:
print("mode=baseline")
result = try_patterns(onnx_path, [])
if result["status"] == "ok":
print(f"baseline=OK trt_regions={result['trt_regions']}")
return 0
print(f"baseline=FAIL trt_regions={result.get('trt_regions', 'n/a')}")
print(f"baseline_error={result['error']}")
return 1
def run_pattern_scan(onnx_path: Path) -> int:
print("mode=pattern_scan")
scan_items = [
"concat",
"split",
"resize2d",
"permute_dims",
"reshape",
"expand_dims",
"relu",
"exp",
"atan",
]
serializer_hits = 0
for name in scan_items:
result = try_patterns(onnx_path, [name])
if result["status"] == "ok":
print(f"scan[{name}]=OK trt_regions={result['trt_regions']}")
continue
error = str(result.get("error", ""))
if "Cannot find the name of the constant" in error:
serializer_hits += 1
kind = "SERIALIZER_CONST_NAME"
else:
kind = "OTHER_FAIL"
print(f"scan[{name}]={kind} trt_regions={result.get('trt_regions', 'n/a')}")
print(f"scan_error[{name}]={error}")
print(f"serializer_const_name_hits={serializer_hits}")
return 0
def main() -> int:
parser = argparse.ArgumentParser(description="Minimal YOLO11n TensorRT BYOC serializer repro")
parser.add_argument("--onnx", type=Path, default=DEFAULT_ONNX)
parser.add_argument("--mode", choices=["baseline", "pattern_scan"], default="pattern_scan")
args = parser.parse_args()
ensure_tensorrt_library_path()
print_model_summary(args.onnx)
if args.mode == "baseline":
return run_baseline(args.onnx)
return run_pattern_scan(args.onnx)
if __name__ == "__main__":
raise SystemExit(main())
```
### Repro command
```bash
python repro_yolo11n_byoc.py --mode pattern_scan
```
`--mode baseline` runs the same graph with only the stable base TensorRT patterns, as a control case.
### Observed result
```text
scan[concat]=SERIALIZER_CONST_NAME trt_regions=21
scan_error[concat]=Check failed: (name != constant_names_.end()) is false: Cannot find the name of the constant: metadata["relax.expr.Constant"][0]
scan[split]=SERIALIZER_CONST_NAME trt_regions=44
scan_error[split]=Check failed: (name != constant_names_.end()) is false: Cannot find the name of the constant: metadata["relax.expr.Constant"][0]
scan[resize2d]=OK trt_regions=42
scan[permute_dims]=OK trt_regions=44
```
### Interpretation
- `concat` / `split` reproduce a serializer constant-name failure
- `resize2d` / `permute_dims` do not reproduce that same failure on this model
- In our smaller handcrafted repros, `resize2d` / `permute_dims` still hit unsupported offload/codegen paths
So these currently look like two different TensorRT BYOC issues, not one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4967254249
@tlopex I tested yolo11n-seg with the script used in the previous reply and found that the existing tuple-output remap could change the semantics of existing TupleGetItem indices, so I updated the fix accordingly. The unit tests and several small models now run successfully. I am attaching the patch in this comment because submitting it to upstream TVM or to your private fork is too annoying.
[relax_byoc_tuple_output_fixes.patch](https://github.com/user-attachments/files/30000714/relax_byoc_tuple_output_fixes.patch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4972720122
Thanks! @guest2180 I just updated according to what you said, could you try again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4802440467
Thanks for reporting, will take a look afterwards
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4962940233
I just send a pr #19998 to fix , could you have a look and try whether it works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4824480817
@guest2180 To address the YOLO case in one go rather than op-by-op, could you share the complete list of operators in your model or you think can be used commonly that don't offload to TensorRT? Since I found that we have to write convert for every op for tensortrt that be used, I am not sure that if it's just like this or it's the usual pattern, please correct me if I'm wrong . If you are interested, you are also welcome to discuss with me or contribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4978365695
@tlopex I tested the new patch, and it looks excellent. YOLO-style subgraphs can now be fully offloaded, and the initial latency numbers are comparable to native TensorRT.
I then tried a point-cloud model and found that it is still some distance away from full-graph offload, so I prepared these two additional patches. They include new operator support, onnx changes, runtime changes, and added unit tests. Please feel free to evaluate the risks and merge the changes if they look reasonable. I also attached the script I used to run the point-cloud model.
I do not have other suitable models on hand for further testing right now, so I do not have more feedback beyond this.
[byoc_relax_correctness_full_offload.patch](https://github.com/user-attachments/files/30040189/byoc_relax_correctness_full_offload.patch)
[tensorrt_runtime_stability_cache.patch](https://github.com/user-attachments/files/30040184/tensorrt_runtime_stability_cache.patch)
[test_pointpillars.py](https://github.com/user-attachments/files/30040194/test_pointpillars.py)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4923343891
@tlopex
Update from our side after testing a newer local TVM checkout around commit `ad87f9b`:
The unsupported-op / naming mismatch part appears to be improved. In particular, the earlier TensorRT BYOC failures around `permute_dims` / `image.resize2d` reaching BYOC but then failing as unsupported are no longer the first blocking issue in our setup.
However, when we switch to the official Relax TensorRT partition flow on a YOLO11 segmentation model, we now hit a different blocker earlier in the BYOC pipeline:
Model:
https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n-seg.onnx
Test script:
```
from __future__ import annotations
import argparse
import traceback
from collections import Counter
from pathlib import Path
import onnx
from tvm.relax.backend.contrib.tensorrt import partition_for_tensorrt
from tvm.relax.frontend.onnx import from_onnx
DEFAULT_ONNX = Path("/home/perception/yolo11n_repro/yolo11n-seg.onnx")
DEFAULT_INPUT_NAME = "images"
DEFAULT_INPUT_SHAPE = [1, 3, 640, 640]
def inspect_onnx_model(onnx_path: Path) -> dict[str, object]:
model = onnx.load(str(onnx_path))
return {
"model": model,
"inputs": [
(value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
for value in model.graph.input
],
"outputs": [
(value.name, [dim.dim_value or dim.dim_param for dim in value.type.tensor_type.shape.dim])
for value in model.graph.output
],
"node_count": len(model.graph.node),
"opset": [(item.domain or "ai.onnx", item.version) for item in model.opset_import],
"ops": Counter(node.op_type for node in model.graph.node),
}
def parse_shape(text: str) -> list[int]:
return [int(part.strip()) for part in text.split(",") if part.strip()]
def main() -> int:
parser = argparse.ArgumentParser(
description="Reproduce the official Relax TensorRT partition cycle on yolo11n-seg.onnx"
)
parser.add_argument("--onnx", type=Path, default=DEFAULT_ONNX)
parser.add_argument("--input-name", default=DEFAULT_INPUT_NAME)
parser.add_argument("--input-shape", default="1,3,640,640")
parser.add_argument("--show-trace", action="store_true")
args = parser.parse_args()
input_shape = parse_shape(args.input_shape)
info = inspect_onnx_model(args.onnx)
print(f"onnx={args.onnx}")
print(f"inputs={info['inputs']}")
print(f"outputs={info['outputs']}")
print(f"nodes={info['node_count']}")
print(f"opset={info['opset']}")
print(f"top_ops={info['ops'].most_common(12)}")
print("step=from_onnx")
mod = from_onnx(info["model"], shape_dict={args.input_name: input_shape})
print("from_onnx=ok")
print("step=partition_for_tensorrt")
try:
_ = partition_for_tensorrt(mod)
except Exception as err:
print(f"partition=FAIL type={type(err).__name__}")
print(f"error={err}")
if args.show_trace:
print("traceback_begin")
print(traceback.format_exc())
print("traceback_end")
return 1
print("partition=OK")
return 0
if __name__ == "__main__":
raise SystemExit(main())
```
python repro_yolo11seg_native_cuda.py --opt-level 0 --show-trace
What we see:
1. `from_onnx(...)` succeeds
2. native TVM CUDA compilation/execution succeeds
3. but `tvm.relax.backend.contrib.tensorrt.partition_for_tensorrt(mod)` fails in `MergeCompositeFunctions()` with a cyclic-dependency error
Error:
`InternalError: Check failed: (depgroup != cur_group) is false: A cyclic dependency detected between the groups lv13 and lv77 are in.`
So from our current testing, this looks like progress: the BYOC operator-name / runtime-converter mismatch is no longer the main blocker, but larger YOLO-style graphs now run into a Relax partition/merge issue (likely around `concat` / `split`-heavy subgraph formation) before codegen.
Separately, the plain detection model also exposes another front-end issue:
https://github.com/ultralytics/assets/releases/download/v8.3.0/yolo11n.onnx
This one currently does not get past `from_onnx(...)` in our environment. It fails while converting `Range` into Relax `arange`, with a type mismatch on the dtype argument (`Expected DataType but got ir.PrimType`).
So at this point we are seeing three stages of progress / remaining issues:
- the original BYOC unsupported-op naming problem is improved
- `yolo11n-seg.onnx` now reaches official TensorRT partitioning but fails with a cyclic dependency during `MergeCompositeFunctions`
- `yolo11n.onnx` still fails earlier in the ONNX importer on `Range`
One additional note from our side: for relatively simple unsupported ops such as `resize2d` or some small shape/dim-related glue handling, it is often still practical for downstream users to do small model-specific or project-specific hand-written workarounds. Those cases may not always require a full official TVM-side update immediately.
The cyclic-dependency problem feels different. Once the graph reaches `MergeCompositeFunctions()` and fails with a group-dependency cycle on official partitioning, that is beyond what we can reasonably patch around at the model/project level. It likely needs a more formal fix in the Relax partition / merge pipeline itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "MasterJH5574 (via GitHub)" <gi...@apache.org>.
MasterJH5574 closed issue #19887: [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning
URL: https://github.com/apache/tvm/issues/19887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "guest2180 (via GitHub)" <gi...@apache.org>.
guest2180 commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4975708193
@tlopex I tried and everything seems ok, btw the current merge rules may split otherwise-supported regions to avoid cyclic dependencies, so even if all operators are supported by the BYOC backend, the whole model may still not be offloaded as a single subgraph. I’m not sure how much this affects steady-state inference latency, but it can increase the initial engine build time because multiple engines/subgraphs need to be built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [I] [Bug] [Tensorrt BYOC] permute_dims / image.resize2d fail as unsupported after Relax partitioning [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on issue #19887:
URL: https://github.com/apache/tvm/issues/19887#issuecomment-4976911374
Just updated again, you can see and have a try, and give some feedbacks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org