You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/18 17:13:20 UTC
[GitHub] [spark] dgd-contributor opened a new pull request #34040: [SPARK-36785][PYTHON] fix DataFrame.isin
dgd-contributor opened a new pull request #34040:
URL: https://github.com/apache/spark/pull/34040
### What changes were proposed in this pull request?
Fix DataFrame.isin when DataFrame has NaN value
### Why are the changes needed?
Fix DataFrame.isin when DataFrame has NaN value
``` python
>>> psdf = ps.DataFrame(
... {"a": [None, 2, 3, 4, 5, 6, 7, 8, None], "b": [None, 5, None, 3, 2, 1, None, 0, 0], "c": [1, 5, 1, 3, 2, 1, 1, 0, 0]},
... )
>>> psdf
a b c
0 NaN NaN 1
1 2.0 5.0 5
2 3.0 NaN 1
3 4.0 3.0 3
4 5.0 2.0 2
5 6.0 1.0 1
6 7.0 NaN 1
7 8.0 0.0 0
8 NaN 0.0 0
>>> other = [1, 2, None]
>>> psdf.isin(other)
a b c
0 None None True
1 True None None
2 None None True
3 None None None
4 None True True
5 None True True
6 None None True
7 None None None
8 None None None
>>> psdf.to_pandas().isin(other)
a b c
0 False False True
1 True False False
2 False False True
3 False False False
4 False True True
5 False True True
6 False False True
7 False False False
8 False False False
```
### Does this PR introduce _any_ user-facing change?
After this PR
``` python
>>> psdf = ps.DataFrame(
... {"a": [None, 2, 3, 4, 5, 6, 7, 8, None], "b": [None, 5, None, 3, 2, 1, None, 0, 0], "c": [1, 5, 1, 3, 2, 1, 1, 0, 0]},
... )
>>> psdf
a b c
0 NaN NaN 1
1 2.0 5.0 5
2 3.0 NaN 1
3 4.0 3.0 3
4 5.0 2.0 2
5 6.0 1.0 1
6 7.0 NaN 1
7 8.0 0.0 0
8 NaN 0.0 0
>>> other = [1, 2, None]
>>> psdf.isin(other)
a b c
0 False False True
1 True False False
2 False False True
3 False False False
4 False True True
5 False True True
6 False False True
7 False False False
8 False False False
```
### How was this patch tested?
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923445335
**[Test build #143467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143467/testReport)** for PR 34040 at commit [`962dc11`](https://github.com/apache/spark/commit/962dc11b8008ede1288b5eb319b0843f32109519).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923384532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143463/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923504271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #34040:
URL: https://github.com/apache/spark/pull/34040#discussion_r712591481
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
+ }
+ )
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), expected)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(
+ psdf.isin({"b": [4, 3, 1, 1, None]}), pdf.isin({"b": [4, 3, 1, 1, None]})
+ )
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [False, False, False, False, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [False, False, False, False, False, False],
+ }
+ )
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), expected)
Review comment:
thanks, updated
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
+ }
+ )
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), expected)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(
+ psdf.isin({"b": [4, 3, 1, 1, None]}), pdf.isin({"b": [4, 3, 1, 1, None]})
+ )
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [False, False, False, False, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [False, False, False, False, False, False],
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923384532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143463/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923489145
Thank you @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923490974
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143467/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923486556
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-922341600
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34040:
URL: https://github.com/apache/spark/pull/34040#discussion_r712527043
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
+ }
+ )
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), expected)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(
+ psdf.isin({"b": [4, 3, 1, 1, None]}), pdf.isin({"b": [4, 3, 1, 1, None]})
+ )
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [False, False, False, False, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [False, False, False, False, False, False],
+ }
+ )
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), expected)
Review comment:
I guess you meant:
```suggestion
self.assert_eq(psdf.isin({"b": [4, 3, 1, 1, None]}), expected)
```
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
Review comment:
This fails with pandas <1.2.
Should be:
```suggestion
"a": [True, False, True, True, False, False],
"b": [True, False, False, True, False, True],
"c": [False, False, False, True, False, True],
```
?
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
+ }
+ )
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), expected)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(
+ psdf.isin({"b": [4, 3, 1, 1, None]}), pdf.isin({"b": [4, 3, 1, 1, None]})
+ )
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [False, False, False, False, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [False, False, False, False, False, False],
Review comment:
And the result should be:
```suggestion
"a": [False, False, False, False, False, False],
"b": [True, False, False, True, False, True],
"c": [False, False, False, False, False, False],
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on pull request #34040: [SPARK-36785][PYTHON] fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-922342736
FYI @ueshin , thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923490974
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143467/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dgd-contributor commented on a change in pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
dgd-contributor commented on a change in pull request #34040:
URL: https://github.com/apache/spark/pull/34040#discussion_r712591427
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
Review comment:
Thank you, updated. Sorry about my again mistake!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923414027
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923478401
**[Test build #143467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143467/testReport)** for PR 34040 at commit [`962dc11`](https://github.com/apache/spark/commit/962dc11b8008ede1288b5eb319b0843f32109519).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923445161
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923504242
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47978/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923362659
**[Test build #143463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143463/testReport)** for PR 34040 at commit [`a56c416`](https://github.com/apache/spark/commit/a56c4168ae6ef86368b52a7454e9fe1120f87969).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #34040:
URL: https://github.com/apache/spark/pull/34040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923362659
**[Test build #143463 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143463/testReport)** for PR 34040 at commit [`a56c416`](https://github.com/apache/spark/commit/a56c4168ae6ef86368b52a7454e9fe1120f87969).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923445161
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923445335
**[Test build #143467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143467/testReport)** for PR 34040 at commit [`962dc11`](https://github.com/apache/spark/commit/962dc11b8008ede1288b5eb319b0843f32109519).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923383474
**[Test build #143463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143463/testReport)** for PR 34040 at commit [`a56c416`](https://github.com/apache/spark/commit/a56c4168ae6ef86368b52a7454e9fe1120f87969).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923439345
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47974/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #34040:
URL: https://github.com/apache/spark/pull/34040#discussion_r712592948
##########
File path: python/pyspark/pandas/tests/test_dataframe.py
##########
@@ -1954,6 +1954,41 @@ def test_isin(self):
with self.assertRaisesRegex(TypeError, msg):
psdf.isin(1)
+ pdf = pd.DataFrame(
+ {
+ "a": [4, 2, 3, 4, 8, 6],
+ "b": [1, None, 9, 4, None, 4],
+ "c": [None, 5, None, 3, 2, 1],
+ },
+ )
+ psdf = ps.from_pandas(pdf)
+
+ if LooseVersion(pd.__version__) >= LooseVersion("1.2"):
+ self.assert_eq(psdf.isin([4, 3, 1, 1, None]), pdf.isin([4, 3, 1, 1, None]))
+ else:
+ expected = pd.DataFrame(
+ {
+ "a": [True, False, True, True, False, False],
+ "b": [True, True, False, True, True, True],
+ "c": [True, False, True, True, False, True],
Review comment:
No worries! Thanks for the fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34040: [SPARK-36785][PYTHON] fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-922341600
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923488512
Thanks! merging to master/3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #34040: [SPARK-36785][PYTHON] Fix DataFrame.isin when DataFrame has NaN value
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #34040:
URL: https://github.com/apache/spark/pull/34040#issuecomment-923337888
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org