You are viewing a plain text version of this content. The canonical link for it is here.
Posted to discuss-archive@tvm.apache.org by "siyiweigeHEW (via GitHub)" <gi...@apache.org> on 2026/08/27 21:47:05 UTC
[PR] [Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss from hardcoded threshold [tvm]
siyiweigeHEW opened a new pull request, #20212:
URL: https://github.com/apache/tvm/pull/20212
Fixes: #20184
## Summary
The Relax ONNX frontend `Softplus._impl_v1` passed a hardcoded `threshold`
(`10.0` for float16, `20.0` otherwise) to `relax.op.nn.softplus`. ONNX
`Softplus` is defined as `y = log(exp(x) + 1)` with **no threshold**
attribute, but the topi lowering of `relax.op.nn.softplus` clamps the result
to the linear function `x` once `beta * x` exceeds the threshold:
```python
# python/tvm/topi/nn/elemwise.py
return tvm.tir.Select(b * value > t, value, (1 / b) * tvm.tir.log(1 + tvm.tir.exp(b * value)))
```
## Root cause
The `threshold` is chosen once for every dtype, and `20` is far too small for
float64: `log(exp(x) + 1) = x + log(1 + exp(-x))` is fully representable in
float64 up to `x ≈ 709`, so clamping at `x = 20` drops the
`log(1 + exp(-x))` term. Differential test vs `onnx.reference` showed an
observable absolute error of `1.87e-9` at `x = 20.1`:
| x (float64) | `onnx.reference` | TVM (before) |
|---|---|---|
| 20.0 | 20.000000002061153 | 20.000000002061153 |
| 20.1 | 20.10000000186501 | **20.1** (err 1.87e-9) |
| 25.0 | 25.000000000013888 | **25.0** (err 1.39e-11) |
(For float32/float16 the clamp is numerically invisible in ordinary ranges —
`log(exp(x)+1)` already rounds to `x` for `x ≳ 15` — and near overflow both
onnxruntime and TVM intentionally return a finite value, so the observable,
spec-relevant defect is the float64 case.)
## Fix
Lower ONNX `Softplus` as the numerically stable, threshold-free form:
```python
x = inputs[0]
dtype = x.ty.dtype
return relax.op.add(
relax.op.maximum(x, relax.const(0, dtype)),
relax.op.log(
relax.op.add(
relax.const(1, dtype),
relax.op.exp(relax.op.negative(relax.op.abs(x))),
)
),
)
```
`max(x, 0) + log(1 + exp(-|x|))` equals `log(exp(x) + 1)` mathematically for
every finite `x`: it never overflows (both terms are bounded), it matches
onnxruntime on float32/float16 (including near overflow, where the naive
formula would overflow to `inf`), and it keeps float64 exact across the whole
representable range. The torch frontend keeps using `relax.op.nn.softplus`
with `beta`/`threshold` since PyTorch's `nn.Softplus` genuinely has those
parameters.
## Validation
Differential test: Relax (build + `VirtualMachine`) vs onnxruntime
(float32/float16) and the mathematically exact stable form cross-checked with
`onnx.reference` (float64; onnxruntime's CPU EP has no float64 Softplus), across
3 dtypes × full value range (incl. the old failure region `x ∈ (20, 30]` and
near-overflow) × multiple shapes and opsets.
| Case | Result |
|---|---|
| fp32 regular / threshold-region / near-overflow `[60..100]` | all OK (vs onnxrt) |
| fp64 regular / `x ∈ (19..50]` (old failure region) / `[600..1000]` | all OK, max\|diff\|=0 (vs exact) |
| fp16 regular / threshold-region `[10..12]` | all OK (vs onnxrt) |
| opset 1 / 13 / 22 (fp32) | all OK |
| fp64 `x = 20.1 / 25 / 30` (pre-fix failures) | max\|diff\| = 0.00e+00 |
Total: **19 differential cases, 19 OK, 0 rejected, 0 numeric mismatches**.
In-tree tests `test_softplus_large_values` (float32 up to `x = 80` vs
onnxruntime) and `test_softplus_float64_accuracy` (float64 vs exact, fails
before this fix) were added; the existing `test_unary[Softplus]` still passes.
## Files changed
- `python/tvm/relax/frontend/onnx/onnx_frontend.py` — `Softplus._impl_v1`:
lower numerically stable `max(x, 0) + log(1 + exp(-|x|))` instead of
`relax.op.nn.softplus` with a hardcoded `threshold`.
- `tests/python/relax/test_frontend_onnx.py` — add
`test_softplus_large_values` and `test_softplus_float64_accuracy`.
## Base
Based on current upstream `main` (`ad0a225074`, 2026-08-28) using the current
`x.ty.dtype` API (vs the older `struct_info.dtype`). The single commit
`e5eede1b5d` applies cleanly on `main` with `git am`/`git apply`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org
Re: [PR] [Fix][Relax][Frontend][ONNX] Fix Softplus accuracy loss from hardcoded threshold [tvm]
Posted by "tlopex (via GitHub)" <gi...@apache.org>.
tlopex merged PR #20212:
URL: https://github.com/apache/tvm/pull/20212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: discuss-archive-unsubscribe@tvm.apache.org
For additional commands, e-mail: discuss-archive-help@tvm.apache.org