You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/02 05:15:48 UTC
[GitHub] [spark] xinrong-databricks opened a new pull request #35706: Support `na_action` and Series input correspondence in `Series.map`
xinrong-databricks opened a new pull request #35706:
URL: https://github.com/apache/spark/pull/35706
### What changes were proposed in this pull request?
Support `na_action` and Series input correspondence in `Series.map`
### Why are the changes needed?
To reach parity to pandas API.
### Does this PR introduce _any_ user-facing change?
Yes. `na_action`, Series input correspondence are supported now:
```py
>>> psser = ps.Series(['cat', 'dog', None, 'rabbit'])
# na_action support
>>> psser.map(lambda x : x.upper(), na_action="ignore")
0 CAT
1 DOG
2 None
3 RABBIT
dtype: object
# Series input correspondence support
>>> pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
>>> psser.map(pser_to_apply)
0 one
1 two
2 None
3 four
dtype: object
```
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
itholic commented on pull request #35706:
URL: https://github.com/apache/spark/pull/35706#issuecomment-1057398424
Seems like something went wrong in the test env. Could you re trigger the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r817500767
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
+
+ # dict correspondence
# Currently Koalas doesn't return NaN as pandas does.
Review comment:
nit: Not related to this patch, but would you mind fixing `Koalas` -> `pandas API on Spark` while we're here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #35706:
URL: https://github.com/apache/spark/pull/35706#issuecomment-1062519167
May I ask for help to get it merged? Thanks! @HyukjinKwon @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r819205751
##########
File path: python/pyspark/pandas/series.py
##########
@@ -1045,8 +1048,18 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
2 I am a None
3 I am a rabbit
dtype: object
+
+ To avoid applying the function to missing values (and keep them as NaN)
+ na_action='ignore' can be used:
+
+ >>> s.map('I am a {}'.format, na_action='ignore')
+ 0 I am a cat
+ 1 I am a dog
+ 2 None
+ 3 I am a rabbit
+ dtype: object
Review comment:
Sounds good! Added.
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
Review comment:
No, `na_action` affects "function" correspondence only. Shall we keep the test case to show the parity with pandas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
itholic commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r817500767
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
+
+ # dict correspondence
# Currently Koalas doesn't return NaN as pandas does.
Review comment:
nit: Not related with this patch, but would you fix `Koalas` -> `pandas API on Spark` while we're here ?
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
+
+ # dict correspondence
# Currently Koalas doesn't return NaN as pandas does.
Review comment:
nit: Not related to this patch, but would you fix `Koalas` -> `pandas API on Spark` while we're here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r819205905
##########
File path: python/pyspark/pandas/series.py
##########
@@ -992,8 +993,10 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
Parameters
----------
- arg : function or dict
+ arg : function, dict or pd.Series
Review comment:
Makes sense, a TODO is added for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r818958752
##########
File path: python/pyspark/pandas/series.py
##########
@@ -1045,8 +1048,18 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
2 I am a None
3 I am a rabbit
dtype: object
+
+ To avoid applying the function to missing values (and keep them as NaN)
+ na_action='ignore' can be used:
+
+ >>> s.map('I am a {}'.format, na_action='ignore')
+ 0 I am a cat
+ 1 I am a dog
+ 2 None
+ 3 I am a rabbit
+ dtype: object
Review comment:
We might also want to have an example taking `pd.Series` as `arg`.
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
Review comment:
nit: shall we use `np.nan` instead of `pd.np.nan` while we are here?
##########
File path: python/pyspark/pandas/series.py
##########
@@ -992,8 +993,10 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
Parameters
----------
- arg : function or dict
+ arg : function, dict or pd.Series
Review comment:
If we accept `pd.Series`, we might also want to accept `ps.Series`, which could be another future work.
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
Review comment:
When the correspondence is a `pd.Series`, seems like `na_action` doesn't have any effect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r819201744
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
Review comment:
Good catch! Modifed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r820443333
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,36 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
- self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
+ self.assert_eq(psser.map({}), pser.map({}).replace({np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
+ )
+
+ # function correspondence
+ self.assert_eq(
+ psser.map(lambda x: x.upper(), na_action="ignore"),
+ pser.map(lambda x: x.upper(), na_action="ignore"),
+ )
+
+ def to_upper(string) -> str:
+ return string.upper() if string else ""
+
+ self.assert_eq(
+ psser.map(to_upper), pser.map(to_upper),
+ )
Review comment:
```suggestion
self.assert_eq(psser.map(to_upper), pser.map(to_upper))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #35706:
URL: https://github.com/apache/spark/pull/35706#issuecomment-1057659469
LGTM, Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
ueshin commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r819805935
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
- self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
+ self.assert_eq(psser.map({}), pser.map({}).replace({np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
+ )
+
+ # function correspondence
+ self.assert_eq(
+ psser.map(lambda x: x.upper(), na_action="ignore"),
+ pser.map(lambda x: x.upper(), na_action="ignore"),
+ )
Review comment:
We should also have a test without the argument?
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
Review comment:
Sure, I'm fine with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin closed pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
ueshin closed pull request #35706:
URL: https://github.com/apache/spark/pull/35706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #35706:
URL: https://github.com/apache/spark/pull/35706#issuecomment-1057670831
Pardon me for rebasing to bypass the weird test failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r818317521
##########
File path: python/pyspark/pandas/series.py
##########
@@ -976,9 +976,10 @@ def cov(self, other: "Series", min_periods: Optional[int] = None) -> float:
else:
return sdf.select(F.covar_samp(*sdf.columns)).head(1)[0][0]
- # TODO: arg should support Series
# TODO: NaN and None
Review comment:
We'd better keep it for empty dict `arg` because of its inconsistency with pandas. I made the TODO more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r818316950
##########
File path: python/pyspark/pandas/series.py
##########
@@ -992,8 +993,10 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
Parameters
----------
- arg : function or dict
+ arg : function, dict, or pd.Series
Mapping correspondence.
+ na_action :
+ If ‘ignore’, propagate NA values, without passing them to the mapping correspondence.
Review comment:
Good catch! Modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r820448734
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,36 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
- self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
+ self.assert_eq(psser.map({}), pser.map({}).replace({np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
+ )
+
+ # function correspondence
+ self.assert_eq(
+ psser.map(lambda x: x.upper(), na_action="ignore"),
+ pser.map(lambda x: x.upper(), na_action="ignore"),
+ )
+
+ def to_upper(string) -> str:
+ return string.upper() if string else ""
+
+ self.assert_eq(
+ psser.map(to_upper), pser.map(to_upper),
+ )
Review comment:
Good catch! Thanks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r820443442
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,36 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
- self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
+ self.assert_eq(psser.map({}), pser.map({}).replace({np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
+ )
+
+ # function correspondence
+ self.assert_eq(
+ psser.map(lambda x: x.upper(), na_action="ignore"),
+ pser.map(lambda x: x.upper(), na_action="ignore"),
+ )
+
+ def to_upper(string) -> str:
+ return string.upper() if string else ""
+
+ self.assert_eq(
+ psser.map(to_upper), pser.map(to_upper),
+ )
Review comment:
to make ci happy. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #35706:
URL: https://github.com/apache/spark/pull/35706#issuecomment-1057677245
May I ask for a review when you are free? @HyukjinKwon @ueshin Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r820326056
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
- # Currently Koalas doesn't return NaN as pandas does.
- self.assert_eq(psser.map({}), pser.map({}).replace({pd.np.nan: None}))
+
+ # dict correspondence
+ # Currently pandas API on Spark doesn't return NaN as pandas does.
+ self.assert_eq(psser.map({}), pser.map({}).replace({np.nan: None}))
d = defaultdict(lambda: "abc")
self.assertTrue("abc" in repr(psser.map(d)))
self.assert_eq(psser.map(d), pser.map(d))
+ # series correspondence
+ pser_to_apply = pd.Series(["one", "two", "four"], index=["cat", "dog", "rabbit"])
+ self.assert_eq(psser.map(pser_to_apply), pser.map(pser_to_apply))
+ self.assert_eq(
+ psser.map(pser_to_apply, na_action="ignore"),
+ pser.map(pser_to_apply, na_action="ignore"),
+ )
+
+ # function correspondence
+ self.assert_eq(
+ psser.map(lambda x: x.upper(), na_action="ignore"),
+ pser.map(lambda x: x.upper(), na_action="ignore"),
+ )
Review comment:
Certainly, added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #35706:
URL: https://github.com/apache/spark/pull/35706#issuecomment-1063536527
Thanks! merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r818303909
##########
File path: python/pyspark/pandas/series.py
##########
@@ -992,8 +993,10 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
Parameters
----------
- arg : function or dict
+ arg : function, dict, or pd.Series
Review comment:
```suggestion
arg : function, dict or pd.Series
```
##########
File path: python/pyspark/pandas/series.py
##########
@@ -976,9 +976,10 @@ def cov(self, other: "Series", min_periods: Optional[int] = None) -> float:
else:
return sdf.select(F.covar_samp(*sdf.columns)).head(1)[0][0]
- # TODO: arg should support Series
# TODO: NaN and None
Review comment:
I guess this `TODO` can also be removed
##########
File path: python/pyspark/pandas/series.py
##########
@@ -992,8 +993,10 @@ def map(self, arg: Union[Dict, Callable]) -> "Series":
Parameters
----------
- arg : function or dict
+ arg : function, dict, or pd.Series
Mapping correspondence.
+ na_action :
+ If ‘ignore’, propagate NA values, without passing them to the mapping correspondence.
Review comment:
```suggestion
If `ignore`, propagate NA values, without passing them to the mapping correspondence.
```
super nits: Looks it's c-p from pandas, but I think "`" it more better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #35706: [SPARK-38387][PYTHON] Support `na_action` and Series input correspondence in `Series.map`
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #35706:
URL: https://github.com/apache/spark/pull/35706#discussion_r817571213
##########
File path: python/pyspark/pandas/tests/test_series.py
##########
@@ -1161,13 +1161,29 @@ def test_append(self):
def test_map(self):
pser = pd.Series(["cat", "dog", None, "rabbit"])
psser = ps.from_pandas(pser)
+
+ # dict correspondence
# Currently Koalas doesn't return NaN as pandas does.
Review comment:
Good catch!! Modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org