You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/10 05:08:41 UTC

[GitHub] [spark] itholic opened a new pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

itholic opened a new pull request #32853:
URL: https://github.com/apache/spark/pull/32853


   ### What changes were proposed in this pull request?
   
   This PR fix the wrong behavior of `Index.difference` in pandas APIs on Spark, based on the comment https://github.com/databricks/koalas/pull/1325#discussion_r647889901 and https://github.com/databricks/koalas/pull/1325#discussion_r647890007
   - it couldn't handle the case properly when `self` is `Index` or `MultiIndex` and `other` is `MultiIndex` or `Index`.
   ```python
   >>> midx1 = ps.MultiIndex.from_tuples([('a', 'x', 1), ('b', 'z', 2), ('k', 'z', 3)])
   >>> idx1 = ps.Index([1, 2, 3])
   >>> midx1 = ps.MultiIndex.from_tuples([('a', 'x', 1), ('b', 'z', 2), ('k', 'z', 3)])
   >>> midx1.difference(idx1)
   pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Index.__iter__()` is not implemented. If you want to collect your data as an NumPy array, use 'to_numpy()' instead.
   ```
   - it's collecting the all data into the driver side when the other is list-like objects, especially when the `other` is distributed object such as Series which is very dangerous.
   
   And added the related test cases.
   
   ### Why are the changes needed?
   
   To correct the incompatible behavior with pandas, and to prevent the case which potentially cause the OOM easily.
   
   ```python
   >>> midx1 = ps.MultiIndex.from_tuples([('a', 'x', 1), ('b', 'z', 2), ('k', 'z', 3)])
   >>> idx1 = ps.Index([1, 2, 3])
   >>> midx1 = ps.MultiIndex.from_tuples([('a', 'x', 1), ('b', 'z', 2), ('k', 'z', 3)])
   >>> midx1.difference(idx1)
   MultiIndex([('a', 'x', 1),
               ('b', 'z', 2),
               ('k', 'z', 3)],
              )
   ```
   
   And now it only using the for loop when the `other` is only the case `list`, `set` or `dict`.
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, the previous bug is fixed as described in the above code examples.
   
   
   ### How was this patch tested?
   
   Manually tested with linter and unittest in local, and it might be passed on CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861169620


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139793/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861166222


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44319/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858316127


   **[Test build #139614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139614/testReport)** for PR 32853 at commit [`61a5c92`](https://github.com/apache/spark/commit/61a5c92ab303dd6bb6aa65e24abb644255e4fa15).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861150734


   **[Test build #139793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139793/testReport)** for PR 32853 at commit [`831ceb7`](https://github.com/apache/spark/commit/831ceb7974dafed8ca549d551cae6b4bdfed79fe).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858384277


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858335817


   **[Test build #139614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139614/testReport)** for PR 32853 at commit [`61a5c92`](https://github.com/apache/spark/commit/61a5c92ab303dd6bb6aa65e24abb644255e4fa15).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861180055


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44319/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861177717


   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858347805


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139614/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858352769


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861164151


   **[Test build #139793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139793/testReport)** for PR 32853 at commit [`831ceb7`](https://github.com/apache/spark/commit/831ceb7974dafed8ca549d551cae6b4bdfed79fe).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-860297029


   BTW, let's make a PR to Koalas side too since this is a bug fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858375789


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32853:
URL: https://github.com/apache/spark/pull/32853


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861195978


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44319/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] itholic commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861143618


   Thanks for the review, @HyukjinKwon 
   
   Let me create a PR when this is ready to merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858316127


   **[Test build #139614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139614/testReport)** for PR 32853 at commit [`61a5c92`](https://github.com/apache/spark/commit/61a5c92ab303dd6bb6aa65e24abb644255e4fa15).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861195978


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44319/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861150734


   **[Test build #139793 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139793/testReport)** for PR 32853 at commit [`831ceb7`](https://github.com/apache/spark/commit/831ceb7974dafed8ca549d551cae6b4bdfed79fe).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858384277


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44141/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861169620


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139793/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] itholic edited a comment on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
itholic edited a comment on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-861143618


   Thanks for the review, @HyukjinKwon 
   
   Let me create a PR to Koalas when this is ready to merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32853:
URL: https://github.com/apache/spark/pull/32853#issuecomment-858347805


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139614/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #32853: [SPARK-35683][PYTHON] Fix Index.difference to avoid collect 'other' to driver side

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32853:
URL: https://github.com/apache/spark/pull/32853#discussion_r650452589



##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -1992,6 +1992,16 @@ def difference(self, other: "Index", sort: Optional[bool] = None) -> "Index":
         """
         from pyspark.pandas.indexes.multi import MultiIndex
 
+        if isinstance(other, Index) and not isinstance(self, type(other)):
+            if isinstance(self, MultiIndex):
+                # In case `self` is MultiIndex and `other` is Index,
+                # return MultiIndex without its names.
+                return self.rename([None] * len(self))
+            elif isinstance(self, Index):
+                # In case `self` is Index and `other` is MultiIndex,
+                # return Index without its name.
+                return self.rename(None)
+
         if not is_list_like(other):

Review comment:
       Can we don't depend on the pandas comparison? `is_list_like` in pandas is not designed to handle pandas-on-Spark instances, and I don't know if it's safe to use it against pandas-on-Spark instances.

##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -1992,6 +1992,16 @@ def difference(self, other: "Index", sort: Optional[bool] = None) -> "Index":
         """
         from pyspark.pandas.indexes.multi import MultiIndex
 
+        if isinstance(other, Index) and not isinstance(self, type(other)):

Review comment:
       can you extract this condition and define it in a variable? It's difficult to follow the if-elses here.

##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2000,11 +2010,16 @@ def difference(self, other: "Index", sort: Optional[bool] = None) -> "Index":
                     sort
                 )
             )
-        # Handling MultiIndex
+        # Handling MultiIndex when `other` is list-like objects.
         if isinstance(self, MultiIndex) and not isinstance(other, MultiIndex):

Review comment:
       can you extract this condition and define it in a variable? It's difficult to follow the if-elses here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org