You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "kenny75557 (via GitHub)" <gi...@apache.org> on 2026/10/11 06:35:07 UTC
[PR] [SPARK-60143][PYTHON][TESTS] Make make_timestamp tests in test_functions independent of the local timezone [spark]
kenny75557 opened a new pull request, #59359:
URL: https://github.com/apache/spark/pull/59359
### What changes were proposed in this pull request?
This PR makes three tests in `pyspark.sql.tests.test_functions` independent of the local timezone:
- `test_try_make_timestamp_ltz`: use the zone ID `"UTC"` instead of `datetime.now().astimezone().tzinfo.__str__()`, which returns
an abbreviation such as `CST` or `PDT`.
- `test_make_timestamp` and `test_try_make_timestamp`: compute the expected values for the `CET` cases from the UTC instant (08:30
UTC) instead of hardcoding `08:30`.
A small helper `_utc_to_local()` converts a UTC wall-clock time to a naive local datetime, which is what `collect()` returns.
### Why are the changes needed?
These tests fail on any machine whose local timezone is not UTC. CI does not catch this because the PySpark jobs run in containers that default to **UTC**.
- Timezone abbreviations are ambiguous for Spark. With `TZ=Asia/Taipei`, Python returns `CST`, which Java reads as US Central
Time, so 10:30 becomes 23:30. With `TZ=America/Los_Angeles` or `America/New_York`, `PDT`/`EDT` are rejected with
`INVALID_TIMEZONE`.
- `collect()` returns timestamps in the local timezone, so the hardcoded `08:30` (10:30 CET in UTC) only matches when the local timezone is UTC.
### Does this PR introduce _any_ user-facing change?
No, test-only change.
### How was this patch tested?
Ran the affected tests under different local timezones, in both classic and Spark Connect mode:
```bash
for tz in UTC Asia/Taipei America/New_York America/Los_Angeles; do
TZ=$tz python/run-tests --testnames "pyspark.sql.tests.test_functions FunctionsTests.test_make_timestamp
FunctionsTests.test_try_make_timestamp FunctionsTests.test_try_make_timestamp_ltz"
done
TZ=Asia/Taipei python/run-tests --testnames "pyspark.sql.tests.connect.test_parity_functions"
```
Before this change, all non-UTC runs fail; after it, all pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Claude Code was used to investigate the root cause and draft the description. I applied the code changes myself, then reviewed and verified them locally under several timezones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60143][PYTHON][TESTS] Make make_timestamp tests in test_functions independent of the local timezone [spark]
Posted by "kenny75557 (via GitHub)" <gi...@apache.org>.
kenny75557 commented on code in PR #59359:
URL: https://github.com/apache/spark/pull/59359#discussion_r4246704342
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -694,8 +700,9 @@ def test_try_make_timestamp(self):
F.try_make_timestamp(timezone=df_dt.timezone)
def test_try_make_timestamp_ltz(self):
- # use local timezone here to avoid flakiness
- data = [(2024, 5, 22, 10, 30, 0, datetime.datetime.now().astimezone().tzinfo.__str__())]
+ # Use an unambiguous zone ID: tz abbreviations such as CST or PDT are misread or
+ # rejected by Spark.
+ data = [(2024, 5, 22, 10, 30, 0, "UTC")]
Review Comment:
Thanks! You're right. Using UTC does change what the test covers. The original intent of passing the local timezone was to keep the expected value environment-independent; the actual bug is that str(tzinfo) yields an abbreviation (e.g. CST, PDT) that Spark misreads as America/Chicago or rejects.
I've update the test to pass the UTC offset on the test date instead, so it keeps the original local-timezone scenario and expected value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60143][PYTHON][TESTS] Make make_timestamp tests in test_functions independent of the local timezone [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #59359:
URL: https://github.com/apache/spark/pull/59359#discussion_r4240901776
##########
python/pyspark/sql/tests/test_functions.py:
##########
@@ -694,8 +700,9 @@ def test_try_make_timestamp(self):
F.try_make_timestamp(timezone=df_dt.timezone)
def test_try_make_timestamp_ltz(self):
- # use local timezone here to avoid flakiness
- data = [(2024, 5, 22, 10, 30, 0, datetime.datetime.now().astimezone().tzinfo.__str__())]
+ # Use an unambiguous zone ID: tz abbreviations such as CST or PDT are misread or
+ # rejected by Spark.
+ data = [(2024, 5, 22, 10, 30, 0, "UTC")]
Review Comment:
Is this necessary? This is changing the test behavior right? You are using UTC now instead of local timezone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org