You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "arzavj (via GitHub)" <gi...@apache.org> on 2024/02/28 02:38:14 UTC
[PR] [SPARK-47202][PySpark] Fix typo breaking datetimes with tzinfo [spark]
arzavj opened a new pull request, #45301:
URL: https://github.com/apache/spark/pull/45301
### What changes were proposed in this pull request?
This PR fixes a bug caused due to a typo.
### Why are the changes needed?
This bug is preventing users from having datetime.datetime objects with tzinfo when using the `TimestampType`
### Does this PR introduce _any_ user-facing change?
No, just a bug fix.
### How was this patch tested?
There ought to be CI that lints code and catches these simple errors at the time of opening the PR.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PySpark] Fix typo breaking datetimes with tzinfo [spark]
Posted by "arzavj (via GitHub)" <gi...@apache.org>.
arzavj commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968086734
I did enable it after it failed but I don't know how to re-run the check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "arzavj (via GitHub)" <gi...@apache.org>.
arzavj commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1977729895
@HyukjinKwon do you know when I can expect 3.5.2 to be released to be able to take advantage of this bug fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45301:
URL: https://github.com/apache/spark/pull/45301#discussion_r1505276588
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -993,7 +993,7 @@ def convert_struct(value: Any) -> Any:
def convert_timestamp(value: Any) -> Any:
if isinstance(value, datetime.datetime) and value.tzinfo is not None:
- ts = pd.Timstamp(value)
+ ts = pd.Timestamp(value)
Review Comment:
+1 for the above comment about adding a test coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #45301: [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo
URL: https://github.com/apache/spark/pull/45301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PySpark] Fix typo breaking datetimes with tzinfo [spark]
Posted by "arzavj (via GitHub)" <gi...@apache.org>.
arzavj commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968085914
@zhengruifeng @ueshin could you please review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PySpark] Fix typo breaking datetimes with tzinfo [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968086497
Hi, @arzavj Can you enable the GHA? https://github.com/apache/spark/pull/45301/checks?check_run_id=22059858407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "arzavj (via GitHub)" <gi...@apache.org>.
arzavj commented on code in PR #45301:
URL: https://github.com/apache/spark/pull/45301#discussion_r1505353843
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -993,7 +993,7 @@ def convert_struct(value: Any) -> Any:
def convert_timestamp(value: Any) -> Any:
if isinstance(value, datetime.datetime) and value.tzinfo is not None:
- ts = pd.Timstamp(value)
+ ts = pd.Timestamp(value)
Review Comment:
Yeah I considered adding a test for this but really what we need here is a linter in CI to catch such typos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45301:
URL: https://github.com/apache/spark/pull/45301#discussion_r1505269608
##########
python/pyspark/sql/pandas/types.py:
##########
@@ -993,7 +993,7 @@ def convert_struct(value: Any) -> Any:
def convert_timestamp(value: Any) -> Any:
if isinstance(value, datetime.datetime) and value.tzinfo is not None:
- ts = pd.Timstamp(value)
+ ts = pd.Timestamp(value)
Review Comment:
I am fine as is because it's pretty obvious but would be good to have a test at `python/pyspark/sql/tests/test_arrow.py`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968321173
PTAL: https://github.com/apache/spark/pull/45308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PySpark] Fix typo breaking datetimes with tzinfo [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968087569
Try `git commit -am "ci" --allow-empty` and push once more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968241304
Let me just merge this in and follow up with a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45301:
URL: https://github.com/apache/spark/pull/45301#issuecomment-1968241780
Merged to master and branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org