You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "canyang25 (via GitHub)" <gi...@apache.org> on 2026/10/08 14:15:55 UTC
[PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
canyang25 opened a new pull request, #59298:
URL: https://github.com/apache/spark/pull/59298
### What changes were proposed in this pull request?
Python UDF timestamp handling dropped datetime.fold during local-time conversion in the repeated DST hour, causing fold=1 values to be observed one hour early. Two changes:
- python/pyspark/sql/conversion.py: Arrow UDF input conversion now uses datetime.datetime.fromtimestamp(value.timestamp()) instead of value.astimezone().replace(tzinfo=None), preserving fold.
- python/pyspark/sql/types.py: TimestampType.toInternal (naive datetime path) now uses dt.timestamp() instead of time.mktime(dt.timetuple()), making epoch conversion fold-aware. Aware-datetime handling is unchanged.
### Why are the changes needed?
When clocks go back for DST, one local hour happens twice; Python marks the second occurrence with datetime.fold=1. PySpark dropped fold in these two spots, so a TIMESTAMP in the second occurrence arrived at Python UDFs one hour too early — even an identity UDF changed the value. Since 4.2 made Arrow UDFs the default, every Python UDF reads such timestamps wrongly by default.
### Does this PR introduce _any_ user-facing change?
Yes — bug fix: Python UDFs now receive the correct instant for timestamps in the repeated DST hour. Non-ambiguous timestamps are unaffected.
### How was this patch tested?
New regression test test_udf_timestamp_dst_fold_round_trip in python/pyspark/sql/tests/test_udf.py: with TZ=America/Los_Angeles and spark.sql.session.timeZone=America/Los_Angeles, timestamp'2021-11-07 09:30:00Z' (epoch 1636277400, fold=1) must arrive at a Python UDF as 1636277400 on BOTH the Arrow path (default) and useArrow=False; identity UDF round-trip returns the same instant; control case for the first 01:30 (fold=0) stays 1636273800. Targeted PySpark UDF tests were run and pass.
JIRA: https://issues.apache.org/jira/browse/SPARK-60081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4236503041
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -809,6 +811,45 @@ def noop(x):
self.assertEqual(df.schema[0].dataType.typeName(), "timestamp_ntz")
self.assertEqual(df.first()[0], datetime.datetime(1970, 1, 1, 0, 0))
+ def test_udf_timestamp_dst_fold_round_trip(self):
+ # SPARK-60081: preserve datetime.fold in the repeated DST hour for Python UDFs.
+ tz = "America/Los_Angeles"
+ tz_prev = os.environ.get("TZ", None)
+ try:
+ os.environ["TZ"] = tz
Review Comment:
**Non-blocking (P2):** `os.environ["TZ"]` plus `time.tzset()` only affects this driver process. The UDF bodies and both changed converters run in Python workers, which the JVM starts with its launch-time environment plus `sc.environment`, so they never see America/Los_Angeles. In CI, PySpark jobs set no TZ, so the containers run in UTC and the workers have no repeated hour. All four assertions then also hold with the pre-fix code; I checked the old expressions under TZ=UTC and they give 1636277400/1636273800.
Other TZ-sensitive suites set `cls.sc.environment["TZ"] = tz` (e.g. `goldenutils.setup_timezone`, `test_arrow_map.py`, `test_pandas_udf_scalar.py`). The Connect parity suite (`UDFParityTests`) has no `sc`, so in-process tests of the two converters under a set TZ may be the more reliable signal there. Also check how the inherited test behaves in `ArrowPythonUDFLegacyTests`, which uses the legacy pandas conversion path.
See **Shared repair plan 2** in the review body.
##########
python/pyspark/sql/types.py:
##########
@@ -492,9 +492,7 @@ def needConversion(self) -> bool:
def toInternal(self, dt: datetime.datetime) -> int:
if dt is not None:
- seconds = (
- calendar.timegm(dt.utctimetuple()) if dt.tzinfo else time.mktime(dt.timetuple())
- )
+ seconds = calendar.timegm(dt.utctimetuple()) if dt.tzinfo else dt.timestamp()
Review Comment:
**Blocking (P1):** For a naive datetime, `dt.timestamp()` already includes `dt.microsecond / 1e6`, so `int(seconds) * 1000000 + dt.microsecond` counts the fraction twice. `int()` also truncates toward zero, so every pre-1970 value with non-zero microseconds comes out one second late, and far-future values with large microseconds round up. For example, with TZ=UTC, `datetime(1969, 12, 31, 23, 59, 59, 500000)` used to give `-500000` and now gives `500000`. With TZ=America/Los_Angeles, `datetime(1950, 5, 3, 12, 0, 0, 123000)` moves by +1,000,000 us. The removed `time.mktime(dt.timetuple())` returned whole seconds, so this was exact before.
This backs classic `createDataFrame` and non-Arrow UDF return values, so those timestamps are silently corrupted. Dropping the microseconds before the float conversion (`dt.replace(microsecond=0).timestamp()`) keeps fold and gives integral seconds, but see the separate note about `datetime.min`, which that form does not handle.
See **Shared repair plan 1** in the review body.
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -809,6 +811,45 @@ def noop(x):
self.assertEqual(df.schema[0].dataType.typeName(), "timestamp_ntz")
self.assertEqual(df.first()[0], datetime.datetime(1970, 1, 1, 0, 0))
+ def test_udf_timestamp_dst_fold_round_trip(self):
+ # SPARK-60081: preserve datetime.fold in the repeated DST hour for Python UDFs.
+ tz = "America/Los_Angeles"
+ tz_prev = os.environ.get("TZ", None)
+ try:
+ os.environ["TZ"] = tz
+ time.tzset()
+ with self.sql_conf(
+ {
+ "spark.sql.session.timeZone": tz,
+ "spark.sql.execution.pythonUDF.arrow.enabled": "true",
+ }
+ ):
+ second_fold = self.spark.sql("SELECT timestamp'2021-11-07 09:30:00Z' AS t")
+ first_fold = self.spark.sql("SELECT timestamp'2021-11-07 08:30:00Z' AS t")
+
+ for kwargs in [{}, {"useArrow": False}]:
+ to_epoch = udf(lambda x: int(x.timestamp()), LongType(), **kwargs)
+ identity = udf(lambda x: x, TimestampType(), **kwargs)
+
+ self.assertEqual(1636277400, second_fold.select(to_epoch("t")).first()[0])
Review Comment:
**Nit (P3):** The `convert_timestamp` change also changes what Spark Connect `collect()` returns: a collected TIMESTAMP in the repeated hour now carries fold=1. Every assertion here reads a Long computed in a worker (`to_epoch`) or in the JVM (`unix_timestamp`), so reverting the conversion.py change would not fail anything under `UDFParityTests`. This test already sets the driver TZ, so an assertion such as `self.assertEqual(second_fold.first()[0].timestamp(), 1636277400)` would cover the client-side converter in Connect mode, and `fromInternal` in classic mode.
See **Shared repair plan 2** in the review body.
##########
python/pyspark/sql/conversion.py:
##########
@@ -1429,7 +1429,7 @@ def convert_timestamp(value: Any) -> Any:
# lossless path), so drop any sub-microsecond digits to match classic collect().
if hasattr(value, "to_pydatetime"):
value = value.to_pydatetime(warn=False)
- return value.astimezone().replace(tzinfo=None)
+ return datetime.datetime.fromtimestamp(value.timestamp())
Review Comment:
**Non-blocking (P2):** `datetime.datetime.fromtimestamp(value.timestamp())` sends the instant through a float. Once |epoch seconds| goes past about 2^33 (roughly before 1685 or after 2255), a double can no longer hold microseconds; for example, `3000-01-01 00:00:00.123457Z` comes back as `.123459`. Near the top of the range it can also round across the year limit: with TZ=UTC, `9999-12-31 23:59:59.999999Z` now raises `ValueError: year 10000 is out of range`, and under America/Los_Angeles it becomes `16:00:00`. The previous `astimezone()` path was exact.
This converter backs Arrow-optimized Python UDF inputs and Spark Connect `collect()`/`toLocalIterator()`, so values that classic `collect()` returns exactly can now drift or fail. `TimestampType.fromInternal` avoids this with integer seconds. Something like `datetime.datetime.fromtimestamp(calendar.timegm(value.utctimetuple())).replace(microsecond=value.microsecond)` keeps fold, because `fromtimestamp` of an int sets it, without the precision loss. I checked that this form returns 01:30 fold=1 for 09:30Z and stays exact for 9999-12-31 23:59:59.999999 under UTC.
**Verification:**
- **Regression:** Far-range aware inputs with microseconds (e.g. 3000-01-01 00:00:00.123457Z, 0001-01-02 08:00:00.123457Z, 9999-12-31 23:59:59.999999Z) convert to the exact local wall time, without drift or ValueError, under UTC and America/Los_Angeles.
- **Behavior:** Under America/Los_Angeles, 2021-11-07 09:30:00Z converts to 01:30 fold=1 and 08:30:00Z to 01:30 fold=0, and .timestamp() of each returns the original instant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4236644080
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -809,6 +811,45 @@ def noop(x):
self.assertEqual(df.schema[0].dataType.typeName(), "timestamp_ntz")
self.assertEqual(df.first()[0], datetime.datetime(1970, 1, 1, 0, 0))
+ def test_udf_timestamp_dst_fold_round_trip(self):
+ # SPARK-60081: preserve datetime.fold in the repeated DST hour for Python UDFs.
+ tz = "America/Los_Angeles"
+ tz_prev = os.environ.get("TZ", None)
+ try:
+ os.environ["TZ"] = tz
Review Comment:
Moved the real coverage to in-process tests in test_types.py with a controlled TZ. They fail on the old code. The UDF test now skips on the legacy pandas path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on PR #59298:
URL: https://github.com/apache/spark/pull/59298#issuecomment-6104051372
Thanks @HyukjinKwon . Added OSError to the fallback for the Windows case near the epoch. There's no test, since the C timestamp() can't be patched. I also updated the PR description. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4239808375
##########
python/pyspark/sql/types.py:
##########
@@ -492,10 +518,11 @@ def needConversion(self) -> bool:
def toInternal(self, dt: datetime.datetime) -> int:
if dt is not None:
- seconds = (
- calendar.timegm(dt.utctimetuple()) if dt.tzinfo else time.mktime(dt.timetuple())
- )
- return int(seconds) * 1000000 + dt.microsecond
+ if dt.tzinfo:
+ seconds = calendar.timegm(dt.utctimetuple())
+ else:
+ seconds = _naive_dt_to_epoch_seconds(dt)
Review Comment:
Fixed. DatetimeConverter now uses _naive_dt_to_epoch_seconds for naive values. Covered by test_datetime_converter_preserves_dst_fold and test_lit_equals_create_dataframe_dst_fold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on PR #59298:
URL: https://github.com/apache/spark/pull/59298#issuecomment-6094465541
Thanks @HyukjinKwon , all addressed. datetime.min falls back to time.mktime, TimestampLTZNanosType now shares the helper, and the PR description is updated. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4238701577
##########
python/pyspark/sql/types.py:
##########
@@ -492,10 +518,11 @@ def needConversion(self) -> bool:
def toInternal(self, dt: datetime.datetime) -> int:
if dt is not None:
- seconds = (
- calendar.timegm(dt.utctimetuple()) if dt.tzinfo else time.mktime(dt.timetuple())
- )
- return int(seconds) * 1000000 + dt.microsecond
+ if dt.tzinfo:
+ seconds = calendar.timegm(dt.utctimetuple())
+ else:
+ seconds = _naive_dt_to_epoch_seconds(dt)
Review Comment:
**Nit (P3):** `toInternal` now resolves a naive repeated-hour datetime by its `fold`. The classic Py4J input converter `DatetimeConverter.convert` (types.py:4127-4139) still computes `time.mktime(obj.timetuple())`, and `timetuple()` drops `fold`, so both occurrences map to one instant there.
Classic `F.lit()` and Column comparisons with a Python datetime go through that converter. Spark Connect `F.lit()` (connect/expressions.py:299-301) and classic `createDataFrame` go through `toInternal`.
Under America/Los_Angeles, take `dt = datetime(2021, 11, 7, 1, 30, fold=1)`. Classic `lit(dt)` becomes 08:30Z, but Connect `lit(dt)` and `createDataFrame([(dt,)], "t timestamp")` give 09:30Z. So in classic mode `df.filter(col("t") == dt)` over that DataFrame misses the row, and the same filter returns different rows in classic and Connect. Before this PR, all three paths used the same `mktime` expression and agreed.
Using `_naive_dt_to_epoch_seconds(obj)` for the naive branch of `DatetimeConverter.convert` would keep every path consistent.
**Verification:**
- **Regression:** In classic mode under America/Los_Angeles, filtering a createDataFrame column of a naive 2021-11-07 01:30 fold=1 datetime with an equality against lit() of the same value matches the row, and so does fold=0.
- **Compatibility:** Aware and non-ambiguous naive datetimes passed through classic lit() resolve to the same instants as before the change.
##########
python/pyspark/sql/conversion.py:
##########
@@ -1429,7 +1430,20 @@ def convert_timestamp(value: Any) -> Any:
# lossless path), so drop any sub-microsecond digits to match classic collect().
if hasattr(value, "to_pydatetime"):
value = value.to_pydatetime(warn=False)
- return value.astimezone().replace(tzinfo=None)
+ # SPARK-60081: integer POSIX seconds so fromtimestamp sets fold on the
+ # local wall time, then re-attach microseconds. A float timestamp would
+ # drop microseconds past ~2255 and can overflow at 9999-12-31 under UTC.
Review Comment:
**Nit (P3):** Float seconds start losing microseconds at 2^33 epoch seconds, which is 2242-03-16, not ~2255. ~2255 is 2^53 microseconds, the limit for an integer microsecond count in a double. On CPython 3.12, float round trips are exact just below 2^33 and drift for about half of the samples just above it. The test comment in `test_arrow_convert_timestamp_range_extremes` already says ~2^33. The ~2255 figure came from my earlier comment, sorry about that.
```suggestion
# drop microseconds past ~2242 and can overflow at 9999-12-31 under UTC.
```
**Verification:**
- **Inspection:** The converter comment names a boundary consistent with 2^33 epoch seconds and no longer says ~2255.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4236644651
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -809,6 +811,45 @@ def noop(x):
self.assertEqual(df.schema[0].dataType.typeName(), "timestamp_ntz")
self.assertEqual(df.first()[0], datetime.datetime(1970, 1, 1, 0, 0))
+ def test_udf_timestamp_dst_fold_round_trip(self):
+ # SPARK-60081: preserve datetime.fold in the repeated DST hour for Python UDFs.
+ tz = "America/Los_Angeles"
+ tz_prev = os.environ.get("TZ", None)
+ try:
+ os.environ["TZ"] = tz
+ time.tzset()
+ with self.sql_conf(
+ {
+ "spark.sql.session.timeZone": tz,
+ "spark.sql.execution.pythonUDF.arrow.enabled": "true",
+ }
+ ):
+ second_fold = self.spark.sql("SELECT timestamp'2021-11-07 09:30:00Z' AS t")
+ first_fold = self.spark.sql("SELECT timestamp'2021-11-07 08:30:00Z' AS t")
+
+ for kwargs in [{}, {"useArrow": False}]:
+ to_epoch = udf(lambda x: int(x.timestamp()), LongType(), **kwargs)
+ identity = udf(lambda x: x, TimestampType(), **kwargs)
+
+ self.assertEqual(1636277400, second_fold.select(to_epoch("t")).first()[0])
Review Comment:
Added driver-side .timestamp() assertions for both folds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4236643237
##########
python/pyspark/sql/conversion.py:
##########
@@ -1429,7 +1429,7 @@ def convert_timestamp(value: Any) -> Any:
# lossless path), so drop any sub-microsecond digits to match classic collect().
if hasattr(value, "to_pydatetime"):
value = value.to_pydatetime(warn=False)
- return value.astimezone().replace(tzinfo=None)
+ return datetime.datetime.fromtimestamp(value.timestamp())
Review Comment:
Done, using your suggested integer-second form. Covered by test_arrow_convert_timestamp_range_extremes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4239809704
##########
python/pyspark/sql/conversion.py:
##########
@@ -1429,7 +1430,20 @@ def convert_timestamp(value: Any) -> Any:
# lossless path), so drop any sub-microsecond digits to match classic collect().
if hasattr(value, "to_pydatetime"):
value = value.to_pydatetime(warn=False)
- return value.astimezone().replace(tzinfo=None)
+ # SPARK-60081: integer POSIX seconds so fromtimestamp sets fold on the
+ # local wall time, then re-attach microseconds. A float timestamp would
+ # drop microseconds past ~2255 and can overflow at 9999-12-31 under UTC.
Review Comment:
Fixed, it now says ~2242.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #59298:
URL: https://github.com/apache/spark/pull/59298#issuecomment-6095755470
Thanks @canyang25. The toInternal side looks right: `_naive_dt_to_epoch_seconds` keeps pre-1970 microseconds exact, `datetime.min` goes through the mktime fallback, and both local-time types share it. One gap traces back to my earlier suggestion. The integer-second form I proposed for `convert_timestamp` raises `ValueError: year 0 is out of range` for instants in roughly the first day of 0001-01-01 (for example `TIMESTAMP'0001-01-01 00:00:00'` under UTC), because the local `fromtimestamp` probes the offset a day earlier. The old `astimezone()` path returned those values, so Connect `collect()` and Arrow UDF inputs now fail on them. Falling back to `value.astimezone().replace(tzinfo=None)` on `(OverflowError, ValueError)`, as the toInternal helper does, would fix it. Also, the PR description on GitHub still describes the earlier `dt.timestamp()` / `fromtimestamp(value.timestamp())` approach, so the update may not have saved.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:6094465541","thread_id":"conversation:6094465541","verdict_sha256":"12a6526df95d6060523b1e644a6680422edc34e5bbf73a806263de56014de191"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4237058691
##########
python/pyspark/sql/types.py:
##########
@@ -484,6 +484,29 @@ def __repr__(self) -> str:
return "TimeType(%d)" % (self.precision)
+def _naive_dt_to_epoch_seconds(dt: datetime.datetime) -> int:
+ """Convert a naive local datetime to whole POSIX seconds, honoring ``fold``.
+
+ Shared by :meth:`TimestampType.toInternal` and :meth:`TimestampLTZNanosType.toInternal`
+ so both local-time types agree on the repeated DST hour (SPARK-60081).
+
+ ``dt.replace(microsecond=0).timestamp()`` is used instead of ``time.mktime(dt.timetuple())``
+ so the result honors ``datetime.fold``. Dropping the microseconds first keeps the float
+ integer-valued (and the ``int()`` conversion exact) across the full 0001..9999 range, so
+ the caller can add ``dt.microsecond`` back without the double-counting / truncation that
+ plain ``int(dt.timestamp())`` produces for pre-1970 and far-future values.
+
+ ``datetime.timestamp()`` probes the UTC offset one day earlier on 0001-01-01 and raises
+ ``OverflowError`` ("year 0 is out of range"); fall back to ``time.mktime`` for that
Review Comment:
**Nit (P3):** On CPython, naive `datetime.min.timestamp()` raises `ValueError('year 0 is out of range')`, not `OverflowError`, so the fallback runs through the `ValueError` arm. `datetime.min` also doesn't round-trip: `TimestampType().fromInternal(TimestampType().toInternal(datetime.datetime.min))` raises the same `ValueError`. Wording such as "raises ValueError ('year 0 is out of range'); fall back to time.mktime so toInternal keeps the previous value for these range-edge inputs" would match the behavior.
**Verification:**
- **Inspection:** The docstring's stated trigger and guarantee match the runtime behavior: naive datetime.min.timestamp() raises ValueError, and toInternal(datetime.min) returns the mktime value.
##########
python/pyspark/sql/tests/test_types.py:
##########
@@ -3887,6 +3889,121 @@ def test_row_with_duplicated_fields(self):
self.assertEqual(list(r), [1, 2, 3])
self.assertEqual([v for v in r], [1, 2, 3])
+ # SPARK-60081: naive-datetime handling in TimestampType.toInternal and
+ # TimestampLTZNanosType.toInternal, and in the Arrow-to-row TIMESTAMP converter. The
+ # production code only runs on POSIX systems where ``time.tzset`` works; skip elsewhere.
Review Comment:
**Nit (P3):** The converters under test run on every platform. Only these tests need `time.tzset()`, which is what the `skipUnless` guards check, so "These tests need time.tzset() (POSIX only); skip elsewhere." would be accurate.
In `test_naive_timestamp_to_internal_range_extremes`, "must still round-trip" overstates the check. The assertion only compares `toInternal(datetime.min)` with the `time.mktime` value, and `fromInternal` of that value raises.
The 'pre-PR' wording in that test also won't mean anything after merge; "the previous time.mktime-based path" would hold up.
**Verification:**
- **Inspection:** The test comments match what the decorators and assertions actually require and check.
##########
python/pyspark/sql/types.py:
##########
@@ -484,6 +484,29 @@ def __repr__(self) -> str:
return "TimeType(%d)" % (self.precision)
+def _naive_dt_to_epoch_seconds(dt: datetime.datetime) -> int:
+ """Convert a naive local datetime to whole POSIX seconds, honoring ``fold``.
+
+ Shared by :meth:`TimestampType.toInternal` and :meth:`TimestampLTZNanosType.toInternal`
+ so both local-time types agree on the repeated DST hour (SPARK-60081).
+
+ ``dt.replace(microsecond=0).timestamp()`` is used instead of ``time.mktime(dt.timetuple())``
+ so the result honors ``datetime.fold``. Dropping the microseconds first keeps the float
+ integer-valued (and the ``int()`` conversion exact) across the full 0001..9999 range, so
+ the caller can add ``dt.microsecond`` back without the double-counting / truncation that
+ plain ``int(dt.timestamp())`` produces for pre-1970 and far-future values.
+
+ ``datetime.timestamp()`` probes the UTC offset one day earlier on 0001-01-01 and raises
+ ``OverflowError`` ("year 0 is out of range"); fall back to ``time.mktime`` for that
+ range-edge case so ``datetime.min`` still round-trips.
+ """
+ try:
+ seconds = dt.replace(microsecond=0).timestamp()
Review Comment:
**Blocking (P1):** This call dispatches to subclass overrides. `pandas.Timestamp.timestamp()` treats a naive value as UTC, and `replace()` returns another `pd.Timestamp`, so a naive `pd.Timestamp` is now converted as if its wall time were UTC.
Under `TZ=America/Los_Angeles`, `TimestampType().toInternal(pd.Timestamp('2021-01-01 12:00'))` returns `1609502400000000` (12:00Z). The equivalent stdlib datetime and the old `time.mktime` path both give `1609531200000000` (12:00 PST). Under Asia/Seoul the value is off by 9 hours.
These values reach `toInternal` without type verification on two paths:
- non-Arrow Python UDF return values (`worker.py`);
- Spark Connect `F.lit(...)` and column comparisons such as `col('t') >= pd.Timestamp(...)` (`connect/expressions.py`).
The result is silently wrong data in any non-UTC process.
The `dt.replace(microsecond=0).timestamp()` form came from my earlier suggestion, and the previous revision's `dt.timestamp()` had the same problem. Calling the stdlib implementation explicitly, `datetime.datetime.timestamp(dt.replace(microsecond=0))`, keeps the local interpretation and still honors fold. I checked that it gives `1609531200` here and `1636277400` for a fold=1 `pd.Timestamp` at 2021-11-07 01:30.
**Verification:**
- **Regression:** Under a non-UTC process zone, toInternal of a naive pandas.Timestamp equals toInternal of the equivalent stdlib datetime for both local-time types.
- **Behavior:** Naive repeated-hour values still map fold=0 and fold=1 to the earlier and later instants, for stdlib and pandas inputs.
- **Compatibility:** Existing stdlib naive-datetime expectations for pre-1970 microseconds and datetime.min/max are unchanged.
##########
python/pyspark/sql/conversion.py:
##########
@@ -1429,7 +1430,13 @@ def convert_timestamp(value: Any) -> Any:
# lossless path), so drop any sub-microsecond digits to match classic collect().
if hasattr(value, "to_pydatetime"):
value = value.to_pydatetime(warn=False)
- return value.astimezone().replace(tzinfo=None)
+ # SPARK-60081: route through integer POSIX seconds so fromtimestamp honors
+ # datetime.fold on the local wall time, then re-attach microseconds. Going
+ # through float timestamps would drop microseconds past ~2255 and overflow at
+ # 9999-12-31 under UTC; keeping the microseconds separate avoids both.
+ return datetime.datetime.fromtimestamp(
Review Comment:
**Non-blocking (P2):** For instants in roughly the first day of year 1, the local `fromtimestamp` probes the UTC offset 24 hours earlier and raises `ValueError: year 0 is out of range`.
Under `TZ=UTC`, `0001-01-01 00:00:00Z` and `0001-01-01 12:00:00.000005Z` now raise here. The replaced `value.astimezone().replace(tzinfo=None)` returned `0001-01-01 00:00` and `0001-01-01 12:00:00.000005`. Under America/Los_Angeles, the failing window runs until about `0001-01-02 08:00Z`.
This converter backs Spark Connect `collect()`/`toLocalIterator()` and Arrow Python UDF/UDTF inputs, so a `TIMESTAMP'0001-01-01 00:00:00'` sentinel now fails the whole action on those paths. `test_arrow_convert_timestamp_range_extremes` starts at `0001-01-02 08:00Z`, just past the window, so it doesn't catch this.
The integer-second form I suggested earlier missed this edge, and the previous float form failed the same way. A fallback like the one in `_naive_dt_to_epoch_seconds` would fix it: `value.astimezone().replace(tzinfo=None)` on `(OverflowError, ValueError)`. That restores these values and still keeps fold=1 at 2021-11-07 09:30Z and exact microseconds at 9999-12-31.
**Verification:**
- **Regression:** First-day-of-year-1 instants convert without ValueError and match the previous astimezone() results under UTC and America/Los_Angeles, for both timestamp types.
- **Behavior:** Repeated-hour fold handling and far-range microsecond exactness from the existing converter tests still hold.
##########
python/pyspark/sql/types.py:
##########
@@ -492,9 +492,7 @@ def needConversion(self) -> bool:
def toInternal(self, dt: datetime.datetime) -> int:
if dt is not None:
- seconds = (
- calendar.timegm(dt.utctimetuple()) if dt.tzinfo else time.mktime(dt.timetuple())
- )
+ seconds = calendar.timegm(dt.utctimetuple()) if dt.tzinfo else dt.timestamp()
Review Comment:
Thanks, the double counting is gone and the pre-1970 test covers it. One follow-up on the form I suggested: `dt.replace(microsecond=0).timestamp()` dispatches to `pandas.Timestamp.timestamp()` when a naive `pd.Timestamp` comes in (for example from a non-Arrow UDF return or a Connect `lit`), and pandas reads naive values as UTC. Under America/Los_Angeles, `toInternal(pd.Timestamp('2021-01-01 12:00'))` now lands 8 hours off. Calling the stdlib method explicitly, `datetime.datetime.timestamp(dt.replace(microsecond=0))`, keeps the local interpretation and still honors fold.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4236503037","thread_id":"inline:4236503037","verdict_sha256":"12a6526df95d6060523b1e644a6680422edc34e5bbf73a806263de56014de191"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60081][PYTHON] Preserve datetime.fold for repeated DST-hour timestamps in Python UDFs [spark]
Posted by "canyang25 (via GitHub)" <gi...@apache.org>.
canyang25 commented on code in PR #59298:
URL: https://github.com/apache/spark/pull/59298#discussion_r4236641834
##########
python/pyspark/sql/types.py:
##########
@@ -492,9 +492,7 @@ def needConversion(self) -> bool:
def toInternal(self, dt: datetime.datetime) -> int:
if dt is not None:
- seconds = (
- calendar.timegm(dt.utctimetuple()) if dt.tzinfo else time.mktime(dt.timetuple())
- )
+ seconds = calendar.timegm(dt.utctimetuple()) if dt.tzinfo else dt.timestamp()
Review Comment:
Fixed. The new _naive_dt_to_epoch_seconds helper uses dt.replace(microsecond=0).timestamp(), so microseconds are added once. Covered by test_naive_timestamp_to_internal_pre_1970_microseconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org