You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "itholic (via GitHub)" <gi...@apache.org> on 2023/05/18 06:26:20 UTC

[GitHub] [spark] itholic opened a new pull request, #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

itholic opened a new pull request, #41211:
URL: https://github.com/apache/spark/pull/41211

   ### What changes were proposed in this pull request?
   
   This PR proposes to upgrade pandas to 2.0.0.
   
   ### Why are the changes needed?
   
   To support latest pandas.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Addressed the existing UTs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes. I already created bunch of tickets for enabling tests, so let me handle them in the follow-ups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes(There should be no behavior changes unless it is a major release). I already created bunch of tickets for enabling tests, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for more detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes. I already created bunch of tickets for enabling tests, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for more detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197553147


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   So, I believe these changes would be inevitable .. but should avoid breaking changes. @itholic is working on listing up all these breaking changes, and we will internally discuss first, and let it out to the community.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes. Let me handle the skipped pandas API on Spark tests in the follow-ups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1198937746


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   eh I'm a bit puzzled and wonder if there is a misunderstanding of the language here.
   "There should be no behavior changes unless it is a major release" where does this come from?
   What @gatorsmile said was "we should not remove these API before the major release Spark 4.0" 
   did he mean that functions that are well-functioning but have been removed in pandas version 2.0 should not be removed? He has also written this under the `def append` which was meant to be removed in your previous PR.
   The whole point of the `pandas API on spark` is that it should be as similar as possible to `pandas` and then there must also be some behavior changes, because it has happened in pandas.
   Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with `@pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0")` and you also create a JIRA for that. So that can more people can help upgrade the `pandas API to version 2.0`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197552394


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Yeah, because we depend on calling pandas right away in several API in pandas API on Spark. For example, this one https://github.com/apache/spark/pull/40658/files#diff-3d6777133e0de63c6dd3761a15d631d0c6520b918babf74522b0d95580954e72L908
   
   ```diff
            >>> ser.rename("a").to_frame().set_index("a").index.astype('int64')
   -        Int64Index([1, 2], dtype='int64', name='a')
   +        Index([1, 2], dtype='int64', name='a')
   ```
   
   Pandas returns a `Index` instead instead of `Int64Index` (see also https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#index-can-now-hold-numpy-numeric-dtypes).
   
   There is no `Int64Index` instance anymore in pandas.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes(**There should be no behavior changes unless it is a major release**).
   
   I already created bunch of [tickets](https://issues.apache.org/jira/browse/SPARK-42618) for enabling tests from pandas 2.0.0, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for additional detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes(**There should be no behavior changes unless it is a major release**). I already created bunch of tickets for enabling tests, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for additional detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1198937746


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   eh I'm a bit puzzled and wonder if there is a misunderstanding of the language here.
   "There should be no behavior changes unless it is a major release" where does this come from?
   What @gatorsmile said was "we should not remove these API before the major release Spark 4.0" 
   did he mean that functions that are well-functioning but have been removed in pandas version 2.0 should not be removed? He has also written this under the def append which was meant to be removed in your previous PR.
   The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with `@pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0")` and you also create a JIRA for that. So that can more people can help upgrade the pandas API to version 2.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197517789


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   For the record, we will need some behaviour changes because of pandas 2.0.0 - we have some API that rely on pandas API on Spark. But I am fine with skipping them for now, and work on them separately.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1198937746


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   eh I'm a bit puzzled and wonder if there is a misunderstanding of the language here.
   "There should be no behavior changes unless it is a major release" where does this come from?
   What @gatorsmile said was "we should not remove these API before the major release Spark 4.0" 
   did he mean that functions that are well-functioning but have been removed in pandas version 2.0 should not be removed? He has also written this under the def append which was meant to be removed in your previous PR.
   The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0") and you also create a JIRA for that. Also, can more people help upgrade the pandas API to version 2.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1198937746


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   eh I'm a bit puzzled and wonder if there is a misunderstanding of the language here.
   "There should be no behavior changes unless it is a major release" where does this come from?
   What @gatorsmile said was "we should not remove these API before the major release Spark 4.0" 
   did he mean that functions that are well-functioning but have been removed in pandas version 2.0 should not be removed? He has also written this under the def append which was meant to be removed in your previous PR.
   The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @skiptest and you also create a JIRA for that. Also, can more people help upgrade the pandas API to version 2.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon closed pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0
URL: https://github.com/apache/spark/pull/41211


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197458264


##########
python/pyspark/pandas/tests/test_dataframe.py:
##########
@@ -4026,10 +4030,19 @@ def test_reindex(self):
             psdf.reindex(["A", "B", "C"], columns=["numbers", "2", "3"]).sort_index(),
         )
 
-        self.assert_eq(
-            pdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
-            psdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
-        )
+        # TODO(SPARK-43271): Match behavior with DataFrame.reindex with specifying `index`.
+        if LooseVersion(pd.__version__) >= LooseVersion("2.0.0"):
+            expected_result = ps.DataFrame([1.0, 2.0, 3.0], index=ps.Index(["A", "B", "C"]))
+            expected_result.columns = pd.Index(["numbers"], name="cols")
+            self.assert_eq(
+                psdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
+                expected_result,
+            )
+        else:
+            self.assert_eq(
+                pdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
+                psdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
+            )

Review Comment:
   FYI: I will address the skipped tests like this, since we can't change the behavior for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1199889980


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   > "There should be no behavior changes unless it is a major release" where does this come from?
   
   AFAIK, it appears to be primarily intended to adhere to the "Considerations when breaking APIs" mentioned in [Versioning Policy](https://spark.apache.org/versioning-policy.html). It also serves as a rule followed uniformly across all components other than the pandas API on Spark.
   
   
   > The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   
   Yes, as @HyukjinKwon mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197553147, that's why we need to discuss whether to allow breaking changes for some APIs in the pandas API on Spark exceptional compared to other components.
   
   > Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0") and you also create a JIRA for that. So that can more people can help upgrade the pandas API on spark to version 2.0?
   
   Yes, as I mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197445641, I am currently skipping all failing tests and creating tickets to allow others to participate in the pandas upgrade. The current PR serves as the groundwork for that, and once the PR is completed, anyone who wishes to contribute can help with the pandas upgrade.
   
   After the PR is merged, anyone who wants to contribute to the pandas API on Spark can pick a ticket from the list at SPARK-42618.



##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   > "There should be no behavior changes unless it is a major release" where does this come from?
   
   AFAIK, it appears to be primarily intended to adhere to the "Considerations when breaking APIs" mentioned in [Versioning Policy](https://spark.apache.org/versioning-policy.html). It also serves as a rule followed uniformly across all components other than the pandas API on Spark.
   
   
   > The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   
   Yes, as @HyukjinKwon mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197553147, that's why we need to discuss whether to allow breaking changes for some APIs in the pandas API on Spark exceptional compared to other components.
   
   > Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0") and you also create a JIRA for that. So that can more people can help upgrade the pandas API on spark to version 2.0?
   
   Yes, as I mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197445641, I am currently skipping all failing tests and creating tickets to allow others to participate in the pandas upgrade. The current PR serves as the groundwork for that, and once the PR is completed, anyone who wishes to contribute can help with the pandas upgrade.
   
   After this PR is merged, anyone who wants to contribute to the pandas API on Spark can pick a ticket from the list at SPARK-42618.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1199889980


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   > "There should be no behavior changes unless it is a major release" where does this come from?
   
   AFAIK, it appears to be primarily intended to adhere to the "Considerations when breaking APIs" mentioned in [Versioning Policy](https://spark.apache.org/versioning-policy.html). It also serves as a rule followed uniformly across all components other than the pandas API on Spark.
   
   
   > The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   
   Yes, as @HyukjinKwon mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197553147, that's why we need to discuss whether to allow breaking changes for some APIs in the pandas API on Spark exceptional compared to other components.
   
   > Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0") and you also create a JIRA for that. So that can more people can help upgrade the pandas API on spark to version 2.0?
   
   Yes, as I mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197445641, I am currently skipping all failing tests and creating tickets to allow others to participate in the pandas upgrade. The current PR serves as the groundwork for that, and once the PR is completed, anyone who wishes to contribute can help with the pandas upgrade.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1198937746


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   eh I'm a bit puzzled and wonder if there is a misunderstanding of the language here.
   "There should be no behavior changes unless it is a major release" where does this come from?
   What @gatorsmile said was "we should not remove these API before the major release Spark 4.0" 
   did he mean that functions that are well-functioning but have been removed in pandas version 2.0 should not be removed? He has also written this under the `def append` which was meant to be removed in your previous PR.
   The whole point of the `pandas API on spark` is that it should be as similar as possible to `pandas` and then there must also be some behavior changes, because it has happened in pandas.
   Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with `@pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0")` and you also create a JIRA for that. So that can more people can help upgrade the `pandas API on spark` to version 2.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #41211:
URL: https://github.com/apache/spark/pull/41211#issuecomment-1569193173

   Nice. Thank you, @itholic and all!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark. I will handle the skipped pandas API on Spark tests in the follow-ups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes(There should be no behavior changes unless it is a major release). I already created bunch of tickets for enabling tests, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for additional detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197526532


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Oh... `behaviour changes because of pandas 2.0.0`?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes(**There should be no behavior changes unless it is a major release**). I already created bunch of [tickets](https://issues.apache.org/jira/browse/SPARK-42618) for enabling tests, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for additional detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197455833


##########
python/pyspark/pandas/tests/indexes/test_datetime.py:
##########
@@ -101,6 +99,28 @@ def test_properties(self):
                 self.assert_eq(psidx.day_of_year, pidx.day_of_year)
                 self.assert_eq(psidx.day_of_week, pidx.day_of_week)
 
+        if LooseVersion(pd.__version__) >= LooseVersion("2.0.0"):
+            # TODO(SPARK-42617): Support isocalendar.week and replace it.
+            expected_results = [
+                ps.Index([1]),
+                ps.Index([1, 1, 13]),
+                ps.Index([52, 52, 1]),
+                ps.Index([52, 52, 52]),
+                ps.Index([52, 52, 52]),
+                ps.Index([52, 52, 52]),
+                ps.Index([52, 52, 52]),
+                ps.Index([52, 52, 52]),
+                ps.Index([52, 1, 2]),
+                ps.Index([13, 26, 39]),
+            ]
+            for psidx, expected_result in zip(self.psidxs, expected_results):
+                self.assert_eq(psidx.week, expected_result)
+                self.assert_eq(psidx.weekofyear, expected_result)
+        else:
+            for psidx, pidx in self.idx_pairs:
+                self.assert_eq(psidx.week, pidx.week)
+                self.assert_eq(psidx.weekofyear, pidx.weekofyear)

Review Comment:
   FYI: I will address the skipped tests like this, since we can't change the behavior for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   Let me skip the tests for now, since there are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark. (There should be no behavior changes unless it is a major release)
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark, and I don't want to introduce any behavior changes. I already created bunch of tickets for enabling tests, so let me handle them in the follow-ups.
   
   Read https://github.com/apache/spark/pull/40658#issuecomment-1552517228 for more detail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   There are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark. I will handle the skipped pandas API on Spark tests in the follow-ups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1199889980


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   > "There should be no behavior changes unless it is a major release" where does this come from?
   
   AFAIK, it appears to be primarily intended to adhere to the "Considerations when breaking APIs" mentioned in [Versioning Policy](https://spark.apache.org/versioning-policy.html). It also serves as a rule followed uniformly across all components other than the pandas API on Spark.
   
   
   > The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   
   Yes, as @HyukjinKwon mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197553147, that's why we need to discuss whether to allow breaking changes for some APIs in the pandas API on Spark exceptional compared to other components.
   
   > Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0") and you also create a JIRA for that. So that can more people can help upgrade the pandas API on spark to version 2.0?
   
   Yes, as I mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197445641, I am currently skipping all failing tests and creating tickets to allow others to participate in the pandas upgrade. The current PR serves as the groundwork for that, and once the PR is completed, anyone who wishes to contribute can help with the pandas upgrade.
   
   So after this PR is merged, anyone who wants to contribute to the pandas API on Spark can pick a ticket from the list at SPARK-42618.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #41211:
URL: https://github.com/apache/spark/pull/41211#issuecomment-1567606989

   Merged to master.
   
   Regarding https://pandas.pydata.org/pandas-docs/version/2.0.2/whatsnew/v2.0.2.html, @itholic is working on the list. Separate PR will be created.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1199889980


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   > "There should be no behavior changes unless it is a major release" where does this come from?
   
   AFAIK, it appears to be primarily intended to adhere to the "Considerations when breaking APIs" mentioned in [Versioning Policy](https://spark.apache.org/versioning-policy.html). It also serves as a rule followed uniformly across all components other than the pandas API on Spark.
   
   
   > The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   
   Yes, as @HyukjinKwon mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197553147, that's why we need to discuss whether to allow breaking changes for some APIs in the pandas API on Spark exceptional compared to other components.
   
   > Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with @pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0") and you also create a JIRA for that. So that can more people can help upgrade the pandas API on spark to version 2.0?
   
   Yes, as I mentioned in https://github.com/apache/spark/pull/41211#discussion_r1197445641, I am currently skipping all failing tests and creating tickets to allow others to participate in the pandas upgrade. The current PR serves as the groundwork for that, and once the PR is completed, anyone who wishes to contribute can help with the pandas upgrade :-)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1197445641


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   There are numerous changes introduced in pandas 2.0.0, and most of them are related to behavior changes in the pandas API on Spark.
   
   In the current PR, our primary goal will be to prioritize support for pandas 2.0.0 in PySpark. I will handle the skipped pandas API on Spark tests in the follow-ups.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] itholic commented on pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "itholic (via GitHub)" <gi...@apache.org>.
itholic commented on PR #41211:
URL: https://github.com/apache/spark/pull/41211#issuecomment-1563757051

   @HyukjinKwon @dongjoon-hyun CI passed.
   Also cc @zhengruifeng @ueshin @xinrong-meng FYI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1206226800


##########
python/pyspark/pandas/frame.py:
##########
@@ -1889,6 +1889,8 @@ def items(self) -> Iterator[Tuple[Name, "Series"]]:
         ...    print('label:', label)
         ...    print('content:', content.to_string())
         ...
+        ...  # doctest: +SKIP
+        # TODO(SPARK-43480): Enable SeriesTests.test_iteritems for pandas 2.0.0.

Review Comment:
   It shouldn't be here. This TODO comment will be shown in docstring



##########
python/pyspark/pandas/tests/indexes/test_datetime.py:
##########
@@ -81,8 +86,6 @@ def test_properties(self):
             self.assert_eq(psidx.minute, pidx.minute)
             self.assert_eq(psidx.second, pidx.second)
             self.assert_eq(psidx.microsecond, pidx.microsecond)
-            self.assert_eq(psidx.week, pidx.week)
-            self.assert_eq(psidx.weekofyear, pidx.weekofyear)

Review Comment:
   why did we remove?



##########
python/pyspark/pandas/tests/test_categorical.py:
##########
@@ -371,10 +388,14 @@ def test_groupby_apply_without_shortcut(self):
         def identity(df) -> ps.DataFrame[zip(psdf.columns, psdf.dtypes)]:
             return df
 
-        self.assert_eq(
-            psdf.groupby("a").apply(identity).sort_values(["a", "b"]).reset_index(drop=True),
-            pdf.groupby("a").apply(identity).sort_values(["a", "b"]).reset_index(drop=True),
-        )
+        if LooseVersion(pd.__version__) >= LooseVersion("2.0.0"):

Review Comment:
   ditto



##########
python/pyspark/pandas/tests/test_dataframe.py:
##########
@@ -4026,10 +4030,19 @@ def test_reindex(self):
             psdf.reindex(["A", "B", "C"], columns=["numbers", "2", "3"]).sort_index(),
         )
 
-        self.assert_eq(
-            pdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
-            psdf.reindex(["A", "B", "C"], index=["numbers", "2", "3"]).sort_index(),
-        )
+        # TODO(SPARK-43271): Match behavior with DataFrame.reindex with specifying `index`.
+        if LooseVersion(pd.__version__) >= LooseVersion("2.0.0"):

Review Comment:
   ditto



##########
python/pyspark/pandas/tests/indexes/test_datetime.py:
##########
@@ -101,6 +104,28 @@ def test_properties(self):
                 self.assert_eq(psidx.day_of_year, pidx.day_of_year)
                 self.assert_eq(psidx.day_of_week, pidx.day_of_week)
 
+        if LooseVersion(pd.__version__) >= LooseVersion("2.0.0"):

Review Comment:
   why do we need this now? I thought you're going to address the different results in a separate tickets.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on a diff in pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on code in PR #41211:
URL: https://github.com/apache/spark/pull/41211#discussion_r1198937746


##########
python/pyspark/pandas/tests/data_type_ops/test_date_ops.py:
##########
@@ -61,6 +63,10 @@ def test_add(self):
         for psser in self.pssers:
             self.assertRaises(TypeError, lambda: self.psser + psser)
 
+    @unittest.skipIf(
+        LooseVersion(pd.__version__) >= LooseVersion("2.0.0"),
+        "TODO(SPARK-43571): Enable DateOpsTests.test_sub for pandas 2.0.0.",
+    )

Review Comment:
   eh I'm a bit puzzled and wonder if there is a misunderstanding of the language here.
   "There should be no behavior changes unless it is a major release" where does this come from?
   What @gatorsmile said was "we should not remove these API before the major release Spark 4.0" 
   did he mean that functions that are well-functioning but have been removed in pandas version 2.0 should not be removed? He has also written this under the def append which was meant to be removed in your previous PR.
   The whole point of the pandas API on spark is that it should be as similar as possible to pandas and then there must also be some behavior changes, because it has happened in pandas.
   Are there any other ways than one person having to fix everything in one and the same PR? e.g. that you mark all tests that fail with `@pytest.mark.skip(reason="see JIRA XXXX for updating to pandas 2.0")` and you also create a JIRA for that. Also, can more people help upgrade the pandas API to version 2.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bjornjorgensen commented on pull request #41211: [SPARK-43024][PYTHON] Upgrade pandas to 2.0.0

Posted by "bjornjorgensen (via GitHub)" <gi...@apache.org>.
bjornjorgensen commented on PR #41211:
URL: https://github.com/apache/spark/pull/41211#issuecomment-1567036614

   https://pandas.pydata.org/pandas-docs/version/2.0.2/whatsnew/v2.0.2.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org