You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/23 01:48:01 UTC
[GitHub] [spark] Yikun commented on a diff in pull request #37521: [SPARK-40078][PYTHON][DOCS] Make pyspark.sql.column examples self-contained
Yikun commented on code in PR #37521:
URL: https://github.com/apache/spark/pull/37521#discussion_r952062535
##########
python/pyspark/sql/column.py:
##########
@@ -495,6 +544,17 @@ def dropFields(self, *fieldNames: str) -> "Column":
.. versionadded:: 3.1.0
+ Parameters
+ ----------
+ fieldNames : str or list
Review Comment:
```suggestion
fieldNames : str
```
`list` is not accurate, it's just a positional arguments.
```
>>> df.withColumn('a', df['a'].dropFields('b', 'c')).show()
+--------------+
| a|
+--------------+
|{3, {4, 5, 6}}|
+--------------+
>>> df.withColumn('a', df['a'].dropFields(['b', 'c'])).show()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/yikun/spark/python/pyspark/sql/column.py", line 543, in dropFields
jc = self._jc.dropFields(_to_seq(sc, fieldNames))
File "/Users/yikun/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
File "/Users/yikun/spark/python/pyspark/sql/utils.py", line 190, in deco
return f(*a, **kw)
File "/Users/yikun/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o88.dropFields.
: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String
```
I think below is a good example to describe positional arguments:
https://github.com/apache/spark/blob/af2d2ee11530a98e748518a81925f3f0a2392209/python/pyspark/sql/column.py#L861
##########
python/pyspark/sql/column.py:
##########
@@ -462,6 +499,18 @@ def withField(self, fieldName: str, col: "Column") -> "Column":
.. versionadded:: 3.1.0
+ Parameters
+ ----------
+ fieldName : str
+ a literal value.
+ The result will only be true at a location if any field matches in the Column.
Review Comment:
`col` parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org