You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "viiccwen (via GitHub)" <gi...@apache.org> on 2026/07/10 16:34:04 UTC

[PR] [Fix][Relax][ONNX] Cast BatchNorm params to input dtype [tvm]

viiccwen opened a new pull request, #19979:
URL: https://github.com/apache/tvm/pull/19979

   Fixes #19977.
   
   ONNX `BatchNormalization` allows the input/output tensor dtype, scale/bias dtype, and mean/variance dtype to be separate floating-point type parameters.
   
   For example, a valid ONNX model may use `float16` data with `float32` gamma, beta, mean, and variance tensors.
   
   The Relax `batch_norm` operator currently requires all five input tensors to have the same dtype.  The ONNX frontend previously forwarded the ONNX inputs directly to `relax.nn.batch_norm`, causing import to fail during normalization
   for mixed-dtype ONNX models.
   
   This patch casts the ONNX BatchNormalization parameter tensors (`scale`, `bias`, `mean`, and `var`) to the data tensor dtype before calling Relax `batch_norm`.
   
   This preserves the ONNX output dtype, which follows the input data dtype, while keeping the fix localized to the frontend compatibility layer.
   
   The regression test builds a minimal ONNX BatchNormalization graph with `float16` data and `float32` parameters, imports it through the Relax ONNX frontend, and checks that the generated Relax `batch_norm` call receives same-dtype inputs.
   
   Verification:
   
   - `python -m pytest tests/python/relax/test_frontend_onnx.py::test_batch_norm_mixed_dtype_params tests/python/relax/test_frontend_onnx.py::test_batch_norm_defaults_to_inference_mode -q`
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Fix][Relax][ONNX] Cast BatchNorm params to input dtype [tvm]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #19979:
URL: https://github.com/apache/tvm/pull/19979#discussion_r3560488804


##########
python/tvm/relax/frontend/onnx/onnx_frontend.py:
##########
@@ -3576,6 +3576,11 @@ def _impl_v15(cls, bb, inputs, attr, params):
         epsilon = attr.get("epsilon", 1e-05)
         momentum = attr.get("momentum", 0.9)
         training_mode = attr.get("training_mode", 0)
+        data_dtype = data.ty.dtype
+        scale, bias, mean, var = [
+            relax.op.astype(x, data_dtype) if x.ty.dtype != data_dtype else x
+            for x in [scale, bias, mean, var]
+        ]

Review Comment:
   ![medium](https://www.gstatic.com/codereviewagent/medium-priority.svg)
   
   When the ONNX parameters (`scale`, `bias`, `mean`, `var`) are constants (which is the default when `keep_params_in_input=False`), casting them using `relax.op.astype` introduces unnecessary `astype` call nodes in the Relax AST.
   
   We can optimize this by directly casting the constant's underlying numpy array and recreating a `relax.Constant` with the target data type. This keeps the imported Relax graph cleaner and avoids relying on subsequent constant folding passes to clean up the cast operations.
   
   ```python
           data_dtype = data.ty.dtype
           scale, bias, mean, var = [
               relax.const(x.data.numpy().astype(str(data_dtype)), str(data_dtype))
               if isinstance(x, relax.Constant) and x.ty.dtype != data_dtype
               else relax.op.astype(x, data_dtype)
               if x.ty.dtype != data_dtype
               else x
               for x in [scale, bias, mean, var]
           ]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Fix][Relax][ONNX] Cast BatchNorm params to input dtype [tvm]

Posted by "yongwww (via GitHub)" <gi...@apache.org>.
yongwww closed pull request #19979: [Fix][Relax][ONNX] Cast BatchNorm params to input dtype
URL: https://github.com/apache/tvm/pull/19979


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Fix][Relax][ONNX] Cast BatchNorm params to input dtype [tvm]

Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on PR #19979:
URL: https://github.com/apache/tvm/pull/19979#issuecomment-4942695698

   Hello @tlopex, 
   
   Updated the implementation. (Follow by [ONNX BatchNormalization-15 schema](https://onnx.ai/onnx/operators/onnx__BatchNormalization.html))
   
   The strategy is:
   - Preserve ONNX's dtype groups: X/Y use T, scale/bias use T1, and mean/variance outputs use T2.
   - For the reported mixed-precision case (float16 X with float16/float32 parameters and statistics), run Relax batch_norm in float32.
   - Cast Y back to T and the running mean/variance back to T2.
   - In training mode, float16 inputs also use float32 computation, following the [ONNX requirement](https://github.com/onnx/onnx/blob/1f5f9ef9f9714c12da0adf79e2cbd49a35dab1db/onnx/defs/nn/defs.cc#L1644-L1660) to compute training statistics in float to avoid overflow.
   - Other mixed-dtype combinations are rejected explicitly instead of applying an unspecified general promotion policy. (can be discussed how to handle. : D)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Fix][Relax][ONNX] Cast BatchNorm params to input dtype [tvm]

Posted by "viiccwen (via GitHub)" <gi...@apache.org>.
viiccwen commented on PR #19979:
URL: https://github.com/apache/tvm/pull/19979#issuecomment-4942596706

   > Thanks for fixing! But could you avoid casting the float32 parameters down to the data dtype, as this can lose precision and change the result.
   > 
   > Since Relax batch_norm requires matching dtypes, please promote all five inputs to a common computation dtype (float32 for this case), run batch_norm, and cast Y back to the original data dtype.
   > 
   > The test can verify that the batch_norm inputs are float32 while the final output remains float16 I think
   
   Thx for pointing out! I'll fix it later. : D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Fix][Relax][ONNX] Cast BatchNorm params to input dtype [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex merged PR #19979:
URL: https://github.com/apache/tvm/pull/19979


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org