[jira] [Created] (SPARK-59089) Recover FloatType columns for the transpiler by widening operands to double

["Holden Karau (Jira)" <ji...@apache.org>]

Holden Karau created SPARK-59089:
------------------------------------

             Summary: Recover FloatType columns for the transpiler by widening operands to double
                 Key: SPARK-59089
                 URL: https://issues.apache.org/jira/browse/SPARK-59089
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 4.3.0
            Reporter: Holden Karau


SPARK-55210 made FloatType columns fall back to interpreted Python entirely, because an
expression that stays in FloatType rounds to 24 bits per step where Python computes in double.
That is more conservative than exactness requires: casting each float operand to DoubleType
reproduces CPython exactly, so those columns could keep their lowering.

## Why

FloatType is common in ML feature tables, and the current rule costs every such column its
lowering permanently.

Measured:
- float32 -> Python float -> float32 round-trips **exactly** on 199163/199163 finite bit
  patterns. So the value the interpreted UDF receives *is* the double a widened expression would
  compute with.
- Staying in FloatType diverged from CPython on **20000/20000** random pairs for `(x + y) * y`.
- Casting each operand to DoubleType and computing in double matched CPython on **20000/20000**
  — for a DoubleType declared return type *and* for a FloatType one, because
  `EvaluatePython.makeFromJava` narrows the interpreted double with a single `c.toFloat`.
- Even the saturation case agrees: the double product 2.5373334837038975e44 under one `.toFloat`
  is Infinity, the same as a trailing `cast(..., float)`.

The objection recorded in the current comment — "would hide the rounding for a single operation
but not for a chain of them, and not the overflow at all" — is about declaring a FloatType
*return* type while still computing in FloatType. It does not apply to computing in double.

## How

Admit FloatType to the "fractional" category in `ResolveTranspiledPythonUDFOptions`, and have the
fractional variant wrap each parameter reference in `.cast("double")` — a no-op on a DoubleType
column that `SimplifyCasts` removes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org