You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/13 07:03:00 UTC
[jira] [Updated] (SPARK-59467) Timezone-naive ArrowDtype timestamps are interpreted as UTC instead of the session timezone in pandas conversion
[ https://issues.apache.org/jira/browse/SPARK-59467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-59467:
-----------------------------------
Labels: pull-request-available (was: )
> Timezone-naive ArrowDtype timestamps are interpreted as UTC instead of the session timezone in pandas conversion
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59467
> URL: https://issues.apache.org/jira/browse/SPARK-59467
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Fangchen Li
> Priority: Major
> Labels: pull-request-available
>
> _check_series_convert_timestamps_internal in pyspark.sql.pandas.types only recognizes numpy datetime64 and pd.DatetimeTZDtype columns. A timezone-naive column backed by pd.ArrowDtype (for example, timestamp[us][pyarrow]) falls through unchanged and is interpreted as UTC rather than in the session timezone, so in a non-UTC session the value is shifted by the session's UTC offset:
> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
> ts = [datetime.datetime(2020, 1, 1, 12, 0)]
> numpy_col = pd.Series(ts, dtype="datetime64[ns]")
> arrow_col = pa.array(ts, pa.timestamp("us")).to_pandas(types_mapper=pd.ArrowDtype)
> spark.createDataFrame(pd.DataFrame(\{"t": numpy_col}), "t timestamp").collect()
> # [Row(t=datetime.datetime(2020, 1, 1, 12, 0))] -- correct
> spark.createDataFrame(pd.DataFrame(\{"t": arrow_col}), "t timestamp").collect()
> # [Row(t=datetime.datetime(2020, 1, 1, 4, 0))] -- off by 8 hours
> This affects createDataFrame with a TimestampType schema and pandas UDFs that return ArrowDtype timestamp Series, and it contradicts the documented behavior of spark.sql.session.timeZone for pandas timestamps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org