You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "siyiweigeHEW (via GitHub)" <gi...@apache.org> on 2026/08/27 21:47:05 UTC

[PR] [Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss from hardcoded threshold [tvm]

siyiweigeHEW opened a new pull request, #20212:
URL: https://github.com/apache/tvm/pull/20212

   Fixes: #20184
   ## Summary
   
   The Relax ONNX frontend `Softplus._impl_v1` passed a hardcoded `threshold`
   (`10.0` for float16, `20.0` otherwise) to `relax.op.nn.softplus`. ONNX
   `Softplus` is defined as `y = log(exp(x) + 1)` with **no threshold**
   attribute, but the topi lowering of `relax.op.nn.softplus` clamps the result
   to the linear function `x` once `beta * x` exceeds the threshold:
   
   ```python
   # python/tvm/topi/nn/elemwise.py
   return tvm.tir.Select(b * value > t, value, (1 / b) * tvm.tir.log(1 + tvm.tir.exp(b * value)))
   ```
   
   ## Root cause
   
   The `threshold` is chosen once for every dtype, and `20` is far too small for
   float64: `log(exp(x) + 1) = x + log(1 + exp(-x))` is fully representable in
   float64 up to `x ≈ 709`, so clamping at `x = 20` drops the
   `log(1 + exp(-x))` term. Differential test vs `onnx.reference` showed an
   observable absolute error of `1.87e-9` at `x = 20.1`:
   
   | x (float64) | `onnx.reference` | TVM (before) |
   |---|---|---|
   | 20.0 | 20.000000002061153 | 20.000000002061153 |
   | 20.1 | 20.10000000186501 | **20.1** (err 1.87e-9) |
   | 25.0 | 25.000000000013888 | **25.0** (err 1.39e-11) |
   
   (For float32/float16 the clamp is numerically invisible in ordinary ranges —
   `log(exp(x)+1)` already rounds to `x` for `x ≳ 15` — and near overflow both
   onnxruntime and TVM intentionally return a finite value, so the observable,
   spec-relevant defect is the float64 case.)
   
   ## Fix
   
   Lower ONNX `Softplus` as the numerically stable, threshold-free form:
   
   ```python
   x = inputs[0]
   dtype = x.ty.dtype
   return relax.op.add(
       relax.op.maximum(x, relax.const(0, dtype)),
       relax.op.log(
           relax.op.add(
               relax.const(1, dtype),
               relax.op.exp(relax.op.negative(relax.op.abs(x))),
           )
       ),
   )
   ```
   
   `max(x, 0) + log(1 + exp(-|x|))` equals `log(exp(x) + 1)` mathematically for
   every finite `x`: it never overflows (both terms are bounded), it matches
   onnxruntime on float32/float16 (including near overflow, where the naive
   formula would overflow to `inf`), and it keeps float64 exact across the whole
   representable range. The torch frontend keeps using `relax.op.nn.softplus`
   with `beta`/`threshold` since PyTorch's `nn.Softplus` genuinely has those
   parameters.
   
   ## Validation
   
   Differential test: Relax (build + `VirtualMachine`) vs onnxruntime
   (float32/float16) and the mathematically exact stable form cross-checked with
   `onnx.reference` (float64; onnxruntime's CPU EP has no float64 Softplus), across
   3 dtypes × full value range (incl. the old failure region `x ∈ (20, 30]` and
   near-overflow) × multiple shapes and opsets.
   
   | Case | Result |
   |---|---|
   | fp32 regular / threshold-region / near-overflow `[60..100]` | all OK (vs onnxrt) |
   | fp64 regular / `x ∈ (19..50]` (old failure region) / `[600..1000]` | all OK, max\|diff\|=0 (vs exact) |
   | fp16 regular / threshold-region `[10..12]` | all OK (vs onnxrt) |
   | opset 1 / 13 / 22 (fp32) | all OK |
   | fp64 `x = 20.1 / 25 / 30` (pre-fix failures) | max\|diff\| = 0.00e+00 |
   
   Total: **19 differential cases, 19 OK, 0 rejected, 0 numeric mismatches**.
   In-tree tests `test_softplus_large_values` (float32 up to `x = 80` vs
   onnxruntime) and `test_softplus_float64_accuracy` (float64 vs exact, fails
   before this fix) were added; the existing `test_unary[Softplus]` still passes.
   
   ## Files changed
   
   - `python/tvm/relax/frontend/onnx/onnx_frontend.py` — `Softplus._impl_v1`:
     lower numerically stable `max(x, 0) + log(1 + exp(-|x|))` instead of
     `relax.op.nn.softplus` with a hardcoded `threshold`.
   - `tests/python/relax/test_frontend_onnx.py` — add
     `test_softplus_large_values` and `test_softplus_float64_accuracy`.
   
   ## Base
   
   Based on current upstream `main` (`ad0a225074`, 2026-08-28) using the current
   `x.ty.dtype` API (vs the older `struct_info.dtype`). The single commit
   `e5eede1b5d` applies cleanly on `main` with `git am`/`git apply`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org


Re: [PR] [Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss from hardcoded threshold [tvm]

Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex merged PR #20212:
URL: https://github.com/apache/tvm/pull/20212


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org