You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/08/25 04:04:30 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #37652: [SPARK-40214][PYTHON][SQL] add 'get' to functions
zhengruifeng opened a new pull request, #37652:
URL: https://github.com/apache/spark/pull/37652
### What changes were proposed in this pull request?
expose `get` to dataframe functions
### Why are the changes needed?
for function parity
### Does this PR introduce _any_ user-facing change?
yes, new API
### How was this patch tested?
added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #37652: [SPARK-40214][PYTHON][SQL] add 'get' to functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #37652:
URL: https://github.com/apache/spark/pull/37652#issuecomment-1226844470
Merged to master, thanks @HyukjinKwon for reivew
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37652: [SPARK-40214][PYTHON][SQL] add 'get' to functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37652:
URL: https://github.com/apache/spark/pull/37652#discussion_r954489472
##########
sql/core/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -3958,6 +3958,26 @@ object functions {
ElementAt(column.expr, lit(value).expr)
}
+ /**
+ * Returns element of array at given (0-based) index. If the index points
+ * outside of the array boundaries, then this function returns NULL.
+ *
+ * @group collection_funcs
+ * @since 3.4.0
+ */
+ def get(column: Column, index: Column): Column = withExpr {
+ new Get(column.expr, index.expr)
+ }
+
+ /**
+ * Returns element of array at given (0-based) index. If the index points
+ * outside of the array boundaries, then this function returns NULL.
+ *
+ * @group collection_funcs
+ * @since 3.4.0
+ */
+ def get(column: Column, index: Int): Column = get(column, lit(index))
Review Comment:
Let's probably don't add this (see the comments on the top of this file)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #37652: [SPARK-40214][PYTHON][SQL] add 'get' to functions
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #37652:
URL: https://github.com/apache/spark/pull/37652#discussion_r954490084
##########
python/pyspark/sql/functions.py:
##########
@@ -4845,6 +4849,73 @@ def element_at(col: "ColumnOrName", extraction: Any) -> Column:
return _invoke_function_over_columns("element_at", col, lit(extraction))
+def get(col: "ColumnOrName", index: Union["ColumnOrName", int]) -> Column:
+ """
+ Collection function: Returns element of array at given (0-based) index.
+ If the index points outside of the array boundaries, then this function
+ returns NULL.
+
+ .. versionadded:: 3.4.0
+
+ Parameters
+ ----------
+ col : :class:`~pyspark.sql.Column` or str
+ name of column containing array
+ index : :class:`~pyspark.sql.Column` or str or int
+ index to check for in array
+
+ Notes
+ -----
+ The position is not 1 based, but 0 based index.
+
+ See Also
+ --------
+ :meth:`element_at`
+
+ Examples
+ --------
+ >>> from pyspark.sql.functions import col, get
Review Comment:
Let's remove this line since we're in this module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #37652: [SPARK-40214][PYTHON][SQL] add 'get' to functions
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #37652: [SPARK-40214][PYTHON][SQL] add 'get' to functions
URL: https://github.com/apache/spark/pull/37652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org