You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "javierdejesusda (via GitHub)" <gi...@apache.org> on 2026/06/14 21:57:45 UTC

[PR] [Relax][ONNX] Fix LayerNormalization no-bias zero tensor shape and dtype [tvm]

javierdejesusda opened a new pull request, #19772:
URL: https://github.com/apache/tvm/pull/19772

   ### Root cause
   
   In the ONNX `LayerNormalization` spec the bias `B` is optional; when omitted it should behave as
   zeros shaped and typed like the scale `W`. In `LayerNormalization._impl_v17`, the synthesized zero
   bias instead took its shape from `data.struct_info.shape[1]` (an unrelated data dim) and hardcoded
   `dtype="float32"`. For input `[2, 3, 4, 8]` with scale `[8]` and `axis=-1` this builds a bias of
   shape `(3,)` while gamma is `(8,)`, so `relax.op.nn.layer_norm` raises a size-mismatch
   `InternalError`. The float32 hardcode also breaks fp16/bf16 no-bias models, since gamma, beta, and
   data must share a dtype. PyTorch's `nn.LayerNorm(..., bias=False)` exports exactly this no-bias form.
   
   ### Fix
   
   Derive both the shape and dtype of the synthesized zero bias from the scale, matching the ONNX
   semantics for an omitted `B` and the existing torch frontend
   (`relax.const(np.zeros(shape), x.struct_info.dtype)`):
   
   ```python
   if bias is None:
       bias = relax.const(_np.zeros(gamma_shape, dtype=scale.struct_info.dtype))
   ```
   
   `gamma_shape` and the `_np`/`get_const_tuple` imports are already present. Deriving the dtype from
   the scale (rather than the issue's float32-only suggestion) is what also fixes the fp16/bf16 case.
   
   ### Test plan
   
   Added non-square no-bias regression cases to `test_frontend_onnx.py::test_layer_norm` (the previous
   no-bias case was square, which masked the bug): float32 `[2,3,4,8]`/scale `[8]` and float16 with
   full `check_correctness`, plus a bf16 importer-only case (ORT's CPU provider has no bf16
   LayerNormalization kernel).
   
   Fixes #19691
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Relax][ONNX] Fix LayerNormalization no-bias zero tensor shape and dtype [tvm]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #19772:
URL: https://github.com/apache/tvm/pull/19772#discussion_r3410325148


##########
python/tvm/relax/frontend/onnx/onnx_frontend.py:
##########
@@ -3834,8 +3834,7 @@ def _impl_v17(cls, bb, inputs, attr, params):
         gamma_shape = get_const_tuple(scale.struct_info.shape)
 
         if bias is None:
-            seq_len = data.struct_info.shape[1].value
-            bias = relax.const([0.0] * seq_len, dtype="float32")
+            bias = relax.const(_np.zeros(gamma_shape, dtype=scale.struct_info.dtype))

Review Comment:
   ![high](https://www.gstatic.com/codereviewagent/high-priority.svg)
   
   Using `_np.zeros(gamma_shape, dtype=scale.struct_info.dtype)` will raise a `TypeError: data type 'bfloat16' not understood` in standard NumPy environments when the dtype is `"bfloat16"`, as NumPy does not natively support `bfloat16` without external libraries like `ml_dtypes`.
   
   To prevent this, we should construct the NumPy array with a standard type like `"float32"` and pass the target dtype to `relax.const`, which natively supports `"bfloat16"` and handles the conversion correctly. This also aligns with the pattern mentioned in the PR description.
   
   ```suggestion
               bias = relax.const(_np.zeros(gamma_shape, dtype="float32"), dtype=scale.struct_info.dtype)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Relax][ONNX] Fix LayerNormalization no-bias zero tensor shape and dtype [tvm]

Posted by "javierdejesusda (via GitHub)" <gi...@apache.org>.
javierdejesusda commented on code in PR #19772:
URL: https://github.com/apache/tvm/pull/19772#discussion_r3410348124


##########
python/tvm/relax/frontend/onnx/onnx_frontend.py:
##########
@@ -3834,8 +3834,7 @@ def _impl_v17(cls, bb, inputs, attr, params):
         gamma_shape = get_const_tuple(scale.struct_info.shape)
 
         if bias is None:
-            seq_len = data.struct_info.shape[1].value
-            bias = relax.const([0.0] * seq_len, dtype="float32")
+            bias = relax.const(_np.zeros(gamma_shape, dtype=scale.struct_info.dtype))

Review Comment:
   Confirmed: np.zeros has no native bfloat16/float8, and the relax.const ml_dtypes import only runs after the np.zeros argument is already evaluated, so the dtype has to move onto relax.const. Went with the existing torch frontend form relax.const(np.zeros(shape), x.struct_info.dtype) rather than a float32 intermediate, for consistency with the other frontends. Fixed in d06b125.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Relax][ONNX] Fix LayerNormalization no-bias zero tensor shape and dtype [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex merged PR #19772:
URL: https://github.com/apache/tvm/pull/19772


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org