You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "OmarAzizi (via GitHub)" <gi...@apache.org> on 2026/08/30 20:58:53 UTC
[PR] [Fix][Relax] Validate non-concat axes before fusing parallel matmul biases [tvm]
OmarAzizi opened a new pull request, #20243:
URL: https://github.com/apache/tvm/pull/20243
CombineParallelMatmul fused parallel matmul biases by checking rank alone, not whether their non-concat axes actually matched, so incompatible shapes like [1, 4] and [2, 5] slipped through and crashed in concat.
Added a check that validates the non-concat axes and skips fusing the bias when they don't match.
Fixes #20205.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [Fix][Relax] Validate non-concat axes before fusing parallel matmul biases [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on code in PR #20243:
URL: https://github.com/apache/tvm/pull/20243#discussion_r3909687607
##########
src/relax/transform/combine_parallel_matmul.cc:
##########
@@ -210,6 +217,25 @@ ffi::TypedFunction<ffi::Map<Var, Expr>(ffi::Map<DFPattern, Var>, ffi::Map<Var, E
continue;
}
+ if (branch_info.bias_dim) {
+ std::vector<ffi::Array<PrimExpr>> bias_shapes;
+ bool bias_shape_unknown = false;
+ for (const auto& bias_var : bias) {
+ auto bias_shape_opt = GetTensorType(bias_var)->GetShape();
+ if (!bias_shape_opt) {
+ bias_shape_unknown = true;
+ break;
+ }
+ bias_shapes.push_back(bias_shape_opt.value());
+ }
+ if (bias_shape_unknown) {
+ return ffi::Map<Var, Expr>{};
+ }
+ if (!shapes_compatible_excluding_trailing_axes(bias_shapes, 1)) {
Review Comment:
Please also require each bias’s last dimension to match the corresponding `splits[i].split_size`. For weights `[3,4]`/`[3,5]` and biases `[2,1]`/`[2,1]`, both original adds are valid via broadcasting, but this check passes and produces a `[2,2]` concatenated bias that cannot broadcast to the combined `[2,9]` output. This case should skip fusion, with a regression test added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [Fix][Relax] Validate non-concat axes before fusing parallel matmul biases [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex commented on code in PR #20243:
URL: https://github.com/apache/tvm/pull/20243#discussion_r3927755040
##########
src/relax/transform/combine_parallel_matmul.cc:
##########
@@ -117,19 +117,26 @@ Patterns CreatePatterns(const BranchInfo& branch_info) {
/*! \brief Create a rewriter for the given parallel matmul branches. */
ffi::TypedFunction<ffi::Map<Var, Expr>(ffi::Map<DFPattern, Var>, ffi::Map<Var, Expr>)> GetRewriter(
const Patterns& patterns, const BranchInfo& branch_info, FCheck check) {
- auto batch_dims_compatible = [](size_t rhs_dim, const std::vector<size_t>& indices,
- const std::vector<ffi::Array<PrimExpr>>& rhs_shapes) {
- arith::Analyzer ana;
- for (auto ind : indices) {
- TVM_FFI_ICHECK_EQ(static_cast<int>(rhs_shapes[ind].size()), rhs_dim);
- // -2 for reduction and concat axes
- for (size_t i = 0; i < rhs_dim - 2; ++i) {
- if (!ana->CanProve(rhs_shapes[indices[0]][i] == rhs_shapes[ind][i])) {
- return false;
+ auto shapes_compatible_excluding_trailing_axes =
+ [](const std::vector<ffi::Array<PrimExpr>>& shapes, size_t num_trailing_axes_excluded) {
+ arith::Analyzer ana;
+ size_t ndim = shapes[0].size();
+ for (const auto& shape : shapes) {
+ TVM_FFI_ICHECK_EQ(shape.size(), ndim);
+ for (size_t i = 0; i < ndim - num_trailing_axes_excluded; ++i) {
+ if (!ana->CanProve(shapes[0][i] == shape[i])) {
+ return false;
+ }
+ }
}
- }
- }
- return true;
+ return true;
+ };
+ auto batch_dims_compatible = [&](const std::vector<size_t>& indices,
+ const std::vector<ffi::Array<PrimExpr>>& rhs_shapes) {
+ std::vector<ffi::Array<PrimExpr>> selected;
+ selected.reserve(indices.size());
+ for (size_t ind : indices) selected.push_back(rhs_shapes[ind]);
+ return shapes_compatible_excluding_trailing_axes(selected, 2);
Review Comment:
Please guard against ranks smaller than `num_trailing_axes_excluded` here. Scalar biases are valid broadcast operands, and `std::optional<int>{0}` still enters the bias-fusion branch. For two matmul branches with scalar biases, `ndim` is 0 and this call excludes one trailing axis, so `ndim - num_trailing_axes_excluded` underflows as a `size_t` and the loop accesses a nonexistent dimension. The later `shape[shape.size() - 1]` check has the same issue.
Could this helper return `false` when `ndim < num_trailing_axes_excluded` (and preferably treat rank mismatches as incompatible rather than `ICHECK`), with a regression test using two scalar biases? Such branches should skip bias fusion because rank-0 tensors have no concat axis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [Fix][Relax] Validate non-concat axes before fusing parallel matmul biases [tvm]
Posted by "OmarAzizi (via GitHub)" <gi...@apache.org>.
OmarAzizi commented on code in PR #20243:
URL: https://github.com/apache/tvm/pull/20243#discussion_r3918852572
##########
src/relax/transform/combine_parallel_matmul.cc:
##########
@@ -210,6 +217,25 @@ ffi::TypedFunction<ffi::Map<Var, Expr>(ffi::Map<DFPattern, Var>, ffi::Map<Var, E
continue;
}
+ if (branch_info.bias_dim) {
+ std::vector<ffi::Array<PrimExpr>> bias_shapes;
+ bool bias_shape_unknown = false;
+ for (const auto& bias_var : bias) {
+ auto bias_shape_opt = GetTensorType(bias_var)->GetShape();
+ if (!bias_shape_opt) {
+ bias_shape_unknown = true;
+ break;
+ }
+ bias_shapes.push_back(bias_shape_opt.value());
+ }
+ if (bias_shape_unknown) {
+ return ffi::Map<Var, Expr>{};
+ }
+ if (!shapes_compatible_excluding_trailing_axes(bias_shapes, 1)) {
Review Comment:
Done, I added the check and a new regression test in a c67ceca.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org