You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tian Gao (Jira)" <ji...@apache.org> on 2026/01/03 22:15:00 UTC

[jira] [Created] (SPARK-54890) Allow user to enforce timezone match in conversion

Tian Gao created SPARK-54890:
--------------------------------

             Summary: Allow user to enforce timezone match in conversion
                 Key: SPARK-54890
                 URL: https://issues.apache.org/jira/browse/SPARK-54890
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Tian Gao


Our timestamp conversion is messed up and it's impossible to fix it without seriously breaking backward compatibility. The fundamental issue is that we mix timezone-aware and timezone-naive timestamps together. Python has implicit rules for naive timestamps and we have a session local timezone config. It's not possible to even define a rule that all code can follow.

The only way to resolve this once and for good is to enforce the conversion. Only allow aware timestamps to convert from/to TimestampType and naive timestamps to convert from/to TimestampNTZType.

We can't make this default, not for now, but at least we can give user a mode that can be reasoned.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org