[jira] [Updated] (SPARK-57696) Preserve TIME precision in the PySpark Arrow/pandas type mapping

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-57696:
-----------------------------------
    Labels: pull-request-available  (was: )

> Preserve TIME precision in the PySpark Arrow/pandas type mapping
> ----------------------------------------------------------------
>
>                 Key: SPARK-57696
>                 URL: https://issues.apache.org/jira/browse/SPARK-57696
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Pandas API on Spark
>    Affects Versions: 4.3.0
>            Reporter: Max Gekk
>            Priority: Major
>              Labels: pull-request-available
>
> h2. What
> Carry the {{TimeType(p)}} fractional-second precision {{p}} (in [0, 9]) across the
> PySpark Arrow/pandas type mapping in {{python/pyspark/sql/pandas/types.py}} so that a
> {{TIME(p)}} column round-trips back to the same {{TIME(p)}}, instead of collapsing to the
> default {{TimeType()}} (precision 6).
> This is the PySpark counterpart of SPARK-57661, which fixes the same gap in the JVM
> {{ArrowUtils}} (Spark <-> Arrow) mapping.
> h2. Why
> {{to_arrow_type}} maps {{TimeType}} to {{pa.time64("ns")}}, which carries only a time unit
> and no fractional-second precision field. {{from_arrow_type}} maps any {{is_time64(at)}}
> back to {{TimeType()}}, whose constructor defaults to {{precision = 6}}. As a result the
> declared precision is lost on any Arrow/pandas round-trip ({{TIME(0)}}, {{TIME(3)}},
> {{TIME(9)}}, ... all read back as {{TIME(6)}}), so PySpark Arrow-based transfer
> (createDataFrame from Arrow/pandas, {{mapInArrow}}, {{toPandas}} schema, etc.) silently
> widens or narrows the type label. The stored values are unaffected; this is purely a
> type-fidelity gap.
> PyArrow's {{time64}} logical type only encodes the unit (us/ns) and cannot represent
> precision [0, 9], so the precision must be carried in the Arrow field metadata, mirroring
> the {{SPARK::time::precision}} key the JVM {{ArrowUtils}} uses in SPARK-57661.
> h2. Scope
> * {{python/pyspark/sql/pandas/types.py}}: when building the Arrow field for a {{TimeType(p)}},
> tag it with the precision metadata key {{SPARK::time::precision}}; when reading an Arrow
> {{time64}} field back, read that key to reconstruct {{TimeType(p)}}.
> * Reuse the same metadata key and fallback semantics as the JVM side (SPARK-57661) for
> consistency.
> h2. Behavior on read-back
> * Metadata present and in [0, 9]: reconstruct the exact {{TimeType(p)}}.
> * Metadata absent (foreign Arrow data) or out of [0, 9]: fall back to the default
> {{TimeType()}} (precision 6), preserving today's behavior for non-Spark producers.
> h2. Out of scope
> * Value semantics / rounding: values are carried verbatim; no change to how {{TIME(p)}}
> values are truncated.
> * The JVM {{ArrowUtils}} mapping (handled in SPARK-57661) and the Spark Connect
> proto/converters (separate sub-tasks).
> h2. How tested
> PySpark Arrow conversion tests: round-trip {{TIME(p)}} for {{p}} in {0, 3, 6, 9} preserves
> {{p}}; a {{time64}} field with no precision metadata falls back to {{TIME(6)}}; the
> precision key does not leak into reconstructed field metadata.
> h2. Does this introduce any user-facing change
> No. The TIME data type is gated behind the internal flag {{spark.sql.timeType.enabled}},
> which is off by default in production. With the flag enabled, a {{TIME(p)}} column
> transferred over Arrow/pandas in PySpark retains its declared precision instead of always
> reading back as {{TIME(6)}}. No change to stored values.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org