You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/02/28 06:31:30 UTC

[PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP]Test timestamp with tzinfo in toPandas and createDataFrame [spark]

HyukjinKwon opened a new pull request, #45308:
URL: https://github.com/apache/spark/pull/45308

   ### What changes were proposed in this pull request?
   
   This PR is a follow up of https://github.com/apache/spark/pull/45301 that actually test the change.
   
   ### Why are the changes needed?
   
   To prevent a regression.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No, test-only.
   
   ### How was this patch tested?
   
   Manually ran the tests.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508490288


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   oh



##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   branch-3.5. I will backport



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #45308: [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized
URL: https://github.com/apache/spark/pull/45308


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508490209


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   Maybe, shall we ignore `zoneinfo` related `import` and tests in Python 3.8 and lower, @HyukjinKwon ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #45308:
URL: https://github.com/apache/spark/pull/45308#issuecomment-1968421360

   Merged to master and branch-3.5.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508490159


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   oh I fixed this.



##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   at https://github.com/apache/spark/pull/45324



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508489711


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   `zoneinfo` is added at Python 3.9.
   - https://docs.python.org/3/library/zoneinfo.html
   
   ![Screenshot 2024-02-29 at 21 06 59](https://github.com/apache/spark/assets/9700541/059737a9-5740-4ab8-adbd-febceb468222)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508489711


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   `zoneinfo` is added at Python 3.9.
   
   ![Screenshot 2024-02-29 at 21 06 59](https://github.com/apache/spark/assets/9700541/059737a9-5740-4ab8-adbd-febceb468222)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508489096


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   Oh, `branch-3.5` PyPy 3.8 seems to complain.
   - https://github.com/apache/spark/actions/runs/8077039915/job/22066597081
   ```
   Traceback (most recent call last):
     File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 197, in _run_module_as_main
       return _run_code(code, main_globals, None,
     File "/usr/local/pypy/pypy3.8/lib/pypy3.8/runpy.py", line 87, in _run_code
       exec(code, run_globals)
     File "/__w/spark/spark/python/pyspark/sql/tests/test_arrow.py", line 28, in <module>
       from zoneinfo import ZoneInfo
   ModuleNotFoundError: No module named 'zoneinfo'
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47202][PYTHON][TESTS][FOLLOW-UP] Test timestamp with tzinfo in toPandas and createDataFrame with Arrow optimized [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #45308:
URL: https://github.com/apache/spark/pull/45308#discussion_r1508495882


##########
python/pyspark/sql/tests/test_arrow.py:
##########
@@ -18,10 +18,12 @@
 import datetime
 import os
 import threading
+import calendar
 import time
 import unittest
 from typing import cast
 from collections import namedtuple
+from zoneinfo import ZoneInfo

Review Comment:
   Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org