You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/31 08:45:34 UTC
[GitHub] [spark] itholic opened a new pull request, #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
itholic opened a new pull request, #37739:
URL: https://github.com/apache/spark/pull/37739
### What changes were proposed in this pull request?
This PR proposes to fix the inconsistent behavior for `Index.intersection` function as below:
When `other` is list of tuple, the behavior of pandas API on Spark is difference from pandas.
- pandas API on Spark
```python
>>> psidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> psidx.intersection([(1, 2), (3, 4)]).sort_values()
MultiIndex([], )
```
- pandas
```python
>>> pidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> pidx.intersection([(1, 2), (3, 4)]).sort_values()
Traceback (most recent call last):
...
ValueError: Names should be list-like for a MultiIndex
```
### Why are the changes needed?
To reach parity with pandas.
### Does this PR introduce _any_ user-facing change?
Yes, the behavior of `Index.intersection` is chaged, when the `other` is list of tuple:
- Before
```python
>>> psidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> psidx.intersection([(1, 2), (3, 4)]).sort_values()
MultiIndex([], )
```
- After
```python
>>> psidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> psidx.intersection([(1, 2), (3, 4)]).sort_values()
Traceback (most recent call last):
...
ValueError: Names should be list-like for a MultiIndex
```
### How was this patch tested?
Added a unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37739:
URL: https://github.com/apache/spark/pull/37739#issuecomment-1236609299
Merged to master, thank you @itholic !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
Posted by GitBox <gi...@apache.org>.
itholic commented on PR #37739:
URL: https://github.com/apache/spark/pull/37739#issuecomment-1236571566
> what if the `psidx` itself is a `MultiIndex`?
>
> ```
> >>> psidx
> Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
> >>> psidx.intersection([(1, 2), (3, 4)]).sort_values()
> MultiIndex([], )
> ```
@zhengruifeng Seems like the `psidx` in your example is still using `Index` rather than `MultiIndex` ?
Maybe do you mean:
```python
>>> psmidx
MultiIndex([(1, 2),
(3, 4)],
)
>>> psmidx.intersection([(1, 2), (3, 4)]).sort_values()
MultiIndex([(1, 2),
(3, 4)],
)
```
??
In this case, it works same as pandas as below:
```python
>>> psmidx.to_pandas().intersection([(1, 2), (3, 4)])
MultiIndex([(1, 2),
(3, 4)],
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
URL: https://github.com/apache/spark/pull/37739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
Posted by GitBox <gi...@apache.org>.
itholic commented on PR #37739:
URL: https://github.com/apache/spark/pull/37739#issuecomment-1237523619
Thanks for the review, @zhengruifeng !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37739:
URL: https://github.com/apache/spark/pull/37739#issuecomment-1236607316
got it, `MultiIndex` has its own `intersection` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37739: [SPARK-40265][PS] Fix the inconsistent behavior for Index.intersection.
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37739:
URL: https://github.com/apache/spark/pull/37739#issuecomment-1234230281
what if the `psidx` itself is a `MultiIndex`?
```
>>> psidx
Int64Index([1, 2, 3, 4], dtype='int64', name='Koalas')
>>> psidx.intersection([(1, 2), (3, 4)]).sort_values()
MultiIndex([], )
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org