You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/02 01:35:06 UTC
[GitHub] [spark] bzhaoopenstack opened a new pull request, #37365: [SPARK-39938][python][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
bzhaoopenstack opened a new pull request, #37365:
URL: https://github.com/apache/spark/pull/37365
We need to follow the pandas behavior of prefix/suffix parameter validation in add_prefix/add_suffix.
Now, we force to validate it as a String type. But pandas looks all values which can be formated as String(implement __str__ func). So it's different here.
### What changes were proposed in this pull request?
We support all kind inputs which can be formated as string.
### Why are the changes needed?
As pandas behavior is different with PySpark when we input other types into add_prefix/add_suffix funcs.
PySpark
```
>>> from pyspark import pandas as ps
>>> df = ps.DataFrame({'A': [1, 2, 3, 4], 'B': [3, 4, 5, 6]}, columns=['A', 'B'])
>>> df.add_suffix(666)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/spark/spark/python/pyspark/pandas/frame.py", line 9060, in add_suffix
assert isinstance(suffix, str)
AssertionError
>>> df.add_suffix(True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/spark/spark/python/pyspark/pandas/frame.py", line 9060, in add_suffix
assert isinstance(suffix, str)
AssertionError
```
Pandas: 1.3.X/1.4.X
```
>>> pdf.add_suffix(0.1)
A0.1 B0.1
0 1 3
1 2 4
2 3 5
3 4 6
>>> pdf.add_suffix(True)
ATrue BTrue
0 1 3
1 2 4
2 3 5
3 4 6
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Input any can be stringable input into add_prefix/add_suffix funcs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37365: [SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #37365:
URL: https://github.com/apache/spark/pull/37365#issuecomment-1201973069
cc @xinrong-meng @itholic @zhengruifeng FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] itholic commented on a diff in pull request #37365: [SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
Posted by GitBox <gi...@apache.org>.
itholic commented on code in PR #37365:
URL: https://github.com/apache/spark/pull/37365#discussion_r935228302
##########
python/pyspark/pandas/tests/test_dataframe.py:
##########
@@ -2701,6 +2701,8 @@ def test_add_prefix(self):
pdf = pd.DataFrame({"A": [1, 2, 3, 4], "B": [3, 4, 5, 6]}, index=np.random.rand(4))
psdf = ps.from_pandas(pdf)
self.assert_eq(pdf.add_prefix("col_"), psdf.add_prefix("col_"))
+ self.assert_eq(pdf.add_prefix(1.1), psdf.add_prefix(1.1))
+ self.assert_eq(pdf.add_prefix(True), psdf.add_prefix(True))
Review Comment:
qq: Can we also accept other types? e.g. `TimestampType`, `datetime` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #37365: [SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #37365:
URL: https://github.com/apache/spark/pull/37365#issuecomment-1311073327
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #37365: [SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #37365: [SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
URL: https://github.com/apache/spark/pull/37365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37365: [SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #37365:
URL: https://github.com/apache/spark/pull/37365#issuecomment-1203205122
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org