You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/23 01:48:01 UTC

[GitHub] [spark] Yikun commented on a diff in pull request #37521: [SPARK-40078][PYTHON][DOCS] Make pyspark.sql.column examples self-contained

Yikun commented on code in PR #37521:
URL: https://github.com/apache/spark/pull/37521#discussion_r952062535


##########
python/pyspark/sql/column.py:
##########
@@ -495,6 +544,17 @@ def dropFields(self, *fieldNames: str) -> "Column":
 
         .. versionadded:: 3.1.0
 
+        Parameters
+        ----------
+        fieldNames : str or list

Review Comment:
   ```suggestion
           fieldNames : str
   ```
   
   `list` is not accurate, it's just a positional arguments.
   
   
   ```
   >>> df.withColumn('a', df['a'].dropFields('b', 'c')).show()
   +--------------+
   |             a|
   +--------------+
   |{3, {4, 5, 6}}|
   +--------------+
   
   >>> df.withColumn('a', df['a'].dropFields(['b', 'c'])).show()
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/Users/yikun/spark/python/pyspark/sql/column.py", line 543, in dropFields
       jc = self._jc.dropFields(_to_seq(sc, fieldNames))
     File "/Users/yikun/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
     File "/Users/yikun/spark/python/pyspark/sql/utils.py", line 190, in deco
       return f(*a, **kw)
     File "/Users/yikun/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
   py4j.protocol.Py4JJavaError: An error occurred while calling o88.dropFields.
   : java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String
   ```
   
   I think below is a good example to describe positional arguments:
   https://github.com/apache/spark/blob/af2d2ee11530a98e748518a81925f3f0a2392209/python/pyspark/sql/column.py#L861



##########
python/pyspark/sql/column.py:
##########
@@ -462,6 +499,18 @@ def withField(self, fieldName: str, col: "Column") -> "Column":
 
         .. versionadded:: 3.1.0
 
+        Parameters
+        ----------
+        fieldName : str
+            a literal value.
+            The result will only be true at a location if any field matches in the Column.

Review Comment:
   `col` parameter?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org