You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/04 07:39:31 UTC
[GitHub] [spark] itholic opened a new pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
itholic opened a new pull request #33634:
URL: https://github.com/apache/spark/pull/33634
### What changes were proposed in this pull request?
This PR proposes fixing the `Index.union` to follow the behavior of pandas 1.3.
**NOTE**: We'd better NOT to follow the some behavior of pandas 1.3, since it is too expensive to count the all duplicated values since pandas-on-Spark deals with the large dataset.
For example, if there are duplicate values in index, pandas chooses the number of duplicates from `self` or `other` with more duplicate values.
```python
>>> pidx1 = pd.Index([1, 1, 1, 1, 1, 2, 2])
>>> pidx2 = pd.Index([1, 1, 2, 2, 2, 2, 2])
>>> pidx1.union(pidx2)
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
```
Where as pandas-on-Spark always chooses the duplicates from `self`.
```python
>>> ps_idx1 = ps.Index([1, 1, 1, 1, 1, 2, 2])
>>> ps_idx2 = ps.Index([1, 1, 2, 2, 2, 2, 2])
>>> ps_idx1.union(ps_idx2)
Int64Index([1, 1, 1, 1, 1, 2, 2], dtype='int64')
```
So, if there are more duplicate values in `self`, it behaves the same as pandas, as below.
```python
>>> # pandas-on-Spark
>>> ps_idx1 = ps.Index([1, 1, 1, 1, 2, 3, 3])
>>> ps_idx2 = ps.Index([1, 1, 1, 2, 3, 3])
>>> ps_idx1.union(psidx2)
Int64Index([1, 1, 1, 1, 2, 3, 3], dtype='int64')
>>> # pandas
>>> psidx1.to_pandas().union(psidx2.to_pandas())
Int64Index([1, 1, 1, 1, 2, 3, 3], dtype='int64')
```
If we want to follow the pandas 100%, we need to count and compare all duplicate values in `self` and `other`, which is extremely expensive (also it requires `has_duplicates` operation which internally performs the count operations twice for every case).
### Why are the changes needed?
We should follow the behavior of pandas as much as possible.
### Does this PR introduce _any_ user-facing change?
Yes, the result for some cases will change.
### How was this patch tested?
Fix the unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893092950
**[Test build #142056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142056/testReport)** for PR 33634 at commit [`75e72d5`](https://github.com/apache/spark/commit/75e72d57da7c7760aa309fced4213d88127bd4c9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683913412
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1542,6 +1540,19 @@ def test_union(self):
pidx2.union(pd.Series([1, 2, 3, 4, 3, 4, 3, 4])),
almost=True,
Review comment:
Wouldn't the test now fail with pandas lower than 1.3? We should at least skip or make the tests pass manually in other versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684851850
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Oh, then do we want to use
```python
@unittest.skipIf(
LooseVersion(pd.__version__) < LooseVersion("1.3")
)
```
and remove all the version related logics such as `if LooseVersion(pd.__version__) >= LooseVersion("0.24"):` and `if LooseVersion(pd.__version__) >= LooseVersion("1.0.0")` ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894179815
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684327912
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Hmm, I'm afraid I'm against the change 6ed03e7.
It doesn't test with the actual pandas behavior we are following, so even if pandas changes the behavior in the future, we won't know it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r682859453
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Could you try `sdf_self.union(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))`?
It seems working at least with the example above:
```py
>>> idx1 = ps.Index([1, 1, 2, 2, 2])
>>> idx2 = ps.Index([1, 1, 1, 2, 2])
>>>
>>> idx1.union(idx2)
Int64Index([1, 1, 1, 2, 2, 2], dtype='int64')
>>> idx1.to_pandas().union(idx2.to_pandas())
Int64Index([1, 1, 1, 2, 2, 2], dtype='int64')
```
I haven't tried the other examples and I'm not sure how expensive this is, though.
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Could you try `sdf_self.union(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))`?
It seems working at least with the example above:
```py
>>> ps_idx1 = ps.Index([1, 1, 1, 1, 1, 2, 2])
>>> ps_idx2 = ps.Index([1, 1, 2, 2, 2, 2, 2])
>>> ps_idx1.union(ps_idx2)
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
>>> ps_idx1.to_pandas().union(ps_idx2.to_pandas())
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
```
I haven't tried the other examples and I'm not sure how expensive this is, though.
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Could you try `sdf_self.unionAll(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))`?
It seems working at least with the example above:
```py
>>> ps_idx1 = ps.Index([1, 1, 1, 1, 1, 2, 2])
>>> ps_idx2 = ps.Index([1, 1, 2, 2, 2, 2, 2])
>>> ps_idx1.union(ps_idx2)
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
>>> ps_idx1.to_pandas().union(ps_idx2.to_pandas())
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
```
I haven't tried the other examples and I'm not sure how expensive this is, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894113400
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142145/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894909826
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892523631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892523631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894903536
**[Test build #142202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142202/testReport)** for PR 33634 at commit [`dc8f0e8`](https://github.com/apache/spark/commit/dc8f0e8719e1ae522cf0b6ecc03e913b105bfa20).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class BlockSavedOnDecommissionedBlockManagerException(blockId: BlockId)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892523631
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894894751
**[Test build #142202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142202/testReport)** for PR 33634 at commit [`dc8f0e8`](https://github.com/apache/spark/commit/dc8f0e8719e1ae522cf0b6ecc03e913b105bfa20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892523631
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894907497
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142202/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894113400
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142145/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894907497
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894174129
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894097344
**[Test build #142145 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142145/testReport)** for PR 33634 at commit [`6ed03e7`](https://github.com/apache/spark/commit/6ed03e74e2d23390b07d2713d8d97f704b0d1398).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893114036
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-906844110
Merged to branch-3.2 too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683955183
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2292,9 +2292,7 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
- sdf = sdf_self.union(sdf_other.subtract(sdf_self))
- if isinstance(self, MultiIndex):
- sdf = sdf.drop_duplicates()
+ sdf = sdf_self.unionAll(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))
if sort:
Review comment:
Thanks, @HyukjinKwon !
Seems like they said "dropping duplicate Index values" was a bug, and fixed in this release.
Related discussion was made at https://github.com/pandas-dev/pandas/issues/36289.
Also checked the `sort` and drop duplicates, works the same as pandas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684564305
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Can we remove the old test cases, and only keep the tests only with pandas 1.3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r682859453
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Could you try `sdf_self.unionAll(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))`?
It seems working at least with the example above:
```py
>>> ps_idx1 = ps.Index([1, 1, 1, 1, 1, 2, 2])
>>> ps_idx2 = ps.Index([1, 1, 2, 2, 2, 2, 2])
>>> ps_idx1.union(ps_idx2)
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
>>> ps_idx1.to_pandas().union(ps_idx2.to_pandas())
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
```
I haven't tried the other examples and I'm not sure how expensive this is, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684857398
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Hm .. okay maybe it's a separate to topic to work on/discuss. I am fine with reverting the specific change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894906084
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46714/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892514160
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r682859453
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Could you try `sdf_self.union(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))`?
It seems working at least with the example above:
```py
>>> idx1 = ps.Index([1, 1, 2, 2, 2])
>>> idx2 = ps.Index([1, 1, 1, 2, 2])
>>>
>>> idx1.union(idx2)
Int64Index([1, 1, 1, 2, 2, 2], dtype='int64')
>>> idx1.to_pandas().union(idx2.to_pandas())
Int64Index([1, 1, 1, 2, 2, 2], dtype='int64')
```
I haven't tried the other examples and I'm not sure how expensive this is, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893081860
**[Test build #142056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142056/testReport)** for PR 33634 at commit [`75e72d5`](https://github.com/apache/spark/commit/75e72d57da7c7760aa309fced4213d88127bd4c9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893100166
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892474140
**[Test build #142025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142025/testReport)** for PR 33634 at commit [`b3d8221`](https://github.com/apache/spark/commit/b3d8221ef6ce79e91e2539c614bf89bcf31b9b7e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894920862
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46714/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893122847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33634:
URL: https://github.com/apache/spark/pull/33634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894909826
Merged to master and branch-3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r682429156
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2235,6 +2235,24 @@ def union(
"""
Form the union of two Index objects.
+ .. note:: For duplicated values, pandas chooses the number of duplicates of self or other
+ with more duplicates. But counting all duplicates is very expensive for large data,
+ so pandas-on-Spark always chooses the number of duplicates in self.
Review comment:
Is there any release note in pandas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893101722
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142056/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683955183
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2292,9 +2292,7 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
- sdf = sdf_self.union(sdf_other.subtract(sdf_self))
- if isinstance(self, MultiIndex):
- sdf = sdf.drop_duplicates()
+ sdf = sdf_self.unionAll(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))
if sort:
Review comment:
Thanks, @HyukjinKwon !
Seems like they said "dropping duplicate Index values" was a bug, and fixed in this release.
Related discussion was made at https://github.com/pandas-dev/pandas/issues/36289 and https://github.com/pandas-dev/pandas/issues/31326.
Also checked the `sort` and drop duplicates, works the same as pandas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683915361
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2292,9 +2292,7 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
- sdf = sdf_self.union(sdf_other.subtract(sdf_self))
- if isinstance(self, MultiIndex):
- sdf = sdf.drop_duplicates()
+ sdf = sdf_self.unionAll(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))
if sort:
Review comment:
@itholic, did you refer to this?
> Bug in Index.union() and MultiIndex.union() dropping duplicate Index values when Index was not monotonic or sort was set to False (GH36289, GH31326, GH40862)
It sounds like now it drops duplicates but the codes above in Pr description look keeping the duplicates. Should we also at least check `sort` and drop duplicates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683033391
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2235,6 +2235,24 @@ def union(
"""
Form the union of two Index objects.
+ .. note:: For duplicated values, pandas chooses the number of duplicates of self or other
+ with more duplicates. But counting all duplicates is very expensive for large data,
+ so pandas-on-Spark always chooses the number of duplicates in self.
Review comment:
They mentioned it's bug fix in their release note at https://pandas.pydata.org/pandas-docs/dev/whatsnew/v1.3.0.html#indexing.
Let me try the solution @ueshin commented https://github.com/apache/spark/pull/33634#discussion_r682859453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893081860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683033391
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2235,6 +2235,24 @@ def union(
"""
Form the union of two Index objects.
+ .. note:: For duplicated values, pandas chooses the number of duplicates of self or other
+ with more duplicates. But counting all duplicates is very expensive for large data,
+ so pandas-on-Spark always chooses the number of duplicates in self.
Review comment:
They mentioned it's bug fix in their release note at https://pandas.pydata.org/pandas-docs/dev/whatsnew/v1.3.0.html#indexing.
Let me try the solution @ueshin commented https://github.com/apache/spark/pull/33634#discussion_r682859453
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Let me try. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893081860
**[Test build #142056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142056/testReport)** for PR 33634 at commit [`75e72d5`](https://github.com/apache/spark/commit/75e72d57da7c7760aa309fced4213d88127bd4c9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683917121
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
can we remove this and share one test base?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894179815
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r682859453
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Could you try `sdf_self.union(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))`?
It seems working at least with the example above:
```py
>>> ps_idx1 = ps.Index([1, 1, 1, 1, 1, 2, 2])
>>> ps_idx2 = ps.Index([1, 1, 2, 2, 2, 2, 2])
>>> ps_idx1.union(ps_idx2)
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
>>> ps_idx1.to_pandas().union(ps_idx2.to_pandas())
Int64Index([1, 1, 1, 1, 1, 2, 2, 2, 2, 2], dtype='int64')
```
I haven't tried the other examples and I'm not sure how expensive this is, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684858914
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Sounds good. Just reverted to the previous fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892448561
**[Test build #142025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142025/testReport)** for PR 33634 at commit [`b3d8221`](https://github.com/apache/spark/commit/b3d8221ef6ce79e91e2539c614bf89bcf31b9b7e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892478380
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46536/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684564446
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
We can leverage `unittest.skipIf`. I would prefer to avoid having duplicated test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894078456
**[Test build #142145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142145/testReport)** for PR 33634 at commit [`6ed03e7`](https://github.com/apache/spark/commit/6ed03e74e2d23390b07d2713d8d97f704b0d1398).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894152886
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46657/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894078456
**[Test build #142145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142145/testReport)** for PR 33634 at commit [`6ed03e7`](https://github.com/apache/spark/commit/6ed03e74e2d23390b07d2713d8d97f704b0d1398).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893081860
**[Test build #142056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142056/testReport)** for PR 33634 at commit [`75e72d5`](https://github.com/apache/spark/commit/75e72d57da7c7760aa309fced4213d88127bd4c9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893101722
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142056/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892484469
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142025/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892484469
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142025/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894894751
**[Test build #142202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142202/testReport)** for PR 33634 at commit [`dc8f0e8`](https://github.com/apache/spark/commit/dc8f0e8719e1ae522cf0b6ecc03e913b105bfa20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892448561
**[Test build #142025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142025/testReport)** for PR 33634 at commit [`b3d8221`](https://github.com/apache/spark/commit/b3d8221ef6ce79e91e2539c614bf89bcf31b9b7e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684851850
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Oh, then do we want to use
```python
@unittest.skipIf(
LooseVersion(pd.__version__) < LooseVersion("1.3")
)
```
and remove all the version related logics such as `if LooseVersion(pd.__version__) >= LooseVersion("0.24"):` and `if LooseVersion(pd.__version__) >= LooseVersion("1.0.0")` ??
And do we want to keep the tests with pandas 1.3 only for `test_union` ??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683913412
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1542,6 +1540,19 @@ def test_union(self):
pidx2.union(pd.Series([1, 2, 3, 4, 3, 4, 3, 4])),
almost=True,
Review comment:
Wouldn't the test now fail with pandas lower than 1.3? We should at least skip or make the tests pass manually in other versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-892438497
cc @HyukjinKwon @ueshin @xinrong-databricks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683915361
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2292,9 +2292,7 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
- sdf = sdf_self.union(sdf_other.subtract(sdf_self))
- if isinstance(self, MultiIndex):
- sdf = sdf.drop_duplicates()
+ sdf = sdf_self.unionAll(sdf_other).exceptAll(sdf_self.intersectAll(sdf_other))
if sort:
Review comment:
@itholic, did you refer to this?
> Bug in Index.union() and MultiIndex.union() dropping duplicate Index values when Index was not monotonic or sort was set to False (GH36289, GH31326, GH40862)
It sounds like now it drops duplicates but the codes above look keeping the duplicates. Should we also at least check `sort` and drop duplicates?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r684019127
##########
File path: python/pyspark/pandas/tests/indexes/test_base.py
##########
@@ -1527,10 +1527,8 @@ def test_union(self):
almost=True,
)
+ # Bug in Index.union dropping duplicated values is fixed in pandas 1.3
if LooseVersion(pd.__version__) >= LooseVersion("1.3"):
Review comment:
Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-894918438
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46714/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33634: [SPARK-36369][PYTHON] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33634:
URL: https://github.com/apache/spark/pull/33634#issuecomment-893122847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46567/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #33634: [SPARK-36369] Fix Index.union to follow pandas 1.3
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #33634:
URL: https://github.com/apache/spark/pull/33634#discussion_r683033479
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2293,8 +2311,6 @@ def union(
sdf_self = self._internal.spark_frame.select(self._internal.index_spark_columns)
sdf_other = other_idx._internal.spark_frame.select(other_idx._internal.index_spark_columns)
sdf = sdf_self.union(sdf_other.subtract(sdf_self))
Review comment:
Let me try. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org