You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/10 07:48:49 UTC
[PR] [SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API [spark]
HyukjinKwon opened a new pull request, #57184:
URL: https://github.com/apache/spark/pull/57184
### What changes were proposed in this pull request?
This documents, for every built-in function in the Scala `functions` API (`functions.scala`), the
accepted input type of each argument (`@param`) and the return type of each DataFrame function
(`@return`), using the numeric umbrella where numeric subtypes cast in and `timestamp` for
TIMESTAMP_NTZ in type phrases.
Subtask of SPARK-57999, split out from apache/spark#57079 for reviewability. This PR covers only
the Scala `functions.scala` surface.
### Why are the changes needed?
The Scala `functions` Scaladoc previously had no per-argument type information and no documented
return type for most functions. This fills that gap.
### Does this PR introduce _any_ user-facing change?
Yes, documentation only. No behavioral or API change.
### How was this patch tested?
Scala style/format (scalastyle + scalafmt) and the docs build pass. Types were derived from and
verified against the analyzer's behavior and each expression's `inputTypes` / `dataType`.
Co-authored-by: Isaac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57184: [SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API
URL: https://github.com/apache/spark/pull/57184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #57184:
URL: https://github.com/apache/spark/pull/57184#discussion_r3577103080
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9453,6 +13624,8 @@ object functions {
* default column name `pos` for position, and `col` for elements in the array and `key` and
* `value` for elements in the map unless specified otherwise.
*
+ * @param e
+ * the target column to explode. A column that evaluates to an array or a map.
Review Comment:
`posexplode` (and `posexplode_outer` just below) got a `@param` but no `@return`, whereas the parallel `explode`/`explode_outer` document one ("a column of the element type of the input array, or the key and value columns of the input map"). Worth mirroring `explode`'s `@return` here, plus the added `pos` position column, so the generator family is documented consistently. (Non-blocking.)
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9548,6 +13745,8 @@ object functions {
*
* @group json_funcs
* @since 2.2.0
+ * @return
+ * Returns a column that evaluates to a struct.
Review Comment:
This `@return` ("evaluates to a struct") is accurate for the `StructType` overloads, but the overloads taking `schema: DataType`, `schema: String`, or `schema: Column` also accept `ArrayType` and `MapType` top-level schemas — the description just above even says "a `MapType` ... `StructType` or `ArrayType` with the specified schema", and `JsonToStructs.dataType` is the schema itself.
Verified against a local build:
```
from_json(col, ArrayType(IntegerType)) -> ArrayType(IntegerType,true)
from_json(col, MapType(StringType, IntegerType)) -> MapType(StringType,IntegerType,true)
from_json(col, lit("array<int>")) -> ArrayType(IntegerType,true)
from_json(col, StructType(...)) -> StructType(...) // the only struct case
```
So `from_json(json, "array<int>")` returns an array, not a struct. Consider, on the non-`StructType` overloads, something like:
```suggestion
* Returns a column of the type given by the schema (a struct, array, or map).
```
(Non-blocking. The `StructType` overloads' `@return` is correct as-is.)
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -5904,24 +8598,34 @@ object functions {
* Only allowed at the end of the format string; specifies that 'expr' indicates a negative
* number with wrapping angled brackets.
*
+ * @param e
+ * The input string to convert to a number. A column that evaluates to a string.
+ * @param format
+ * The format to use to convert the value. A column that evaluates to a string. Must be a
+ * constant.
* @group string_funcs
* @since 3.5.0
+ * @return
+ * Returns a column that evaluates to a decimal.
*/
def to_number(e: Column, format: Column): Column = Column.fn("to_number", e, format)
/**
* Replaces all occurrences of `search` with `replace`.
*
* @param src
- * A column of string to be replaced
+ * A column of string to be replaced. A column that evaluates to a string.
* @param search
- * A column of string, If `search` is not found in `str`, `str` is returned unchanged.
+ * A column of string, If `search` is not found in `str`, `str` is returned unchanged. A
Review Comment:
Minor: "A column of string, If `search` is not found ..." has a comma splice (the comma before "If" should be a period) and "of string" should be "of strings". This text is pre-existing, but since this PR edits the line (appends the type sentence), it's cheap to tidy in passing. (Nit.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58086][SQL][DOCS] Document accepted input and return types in the Scala functions API [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57184:
URL: https://github.com/apache/spark/pull/57184#issuecomment-5030897295
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/99025ce39d587125f37bb4305a29c26c0b9317ff
- merged into branch-4.x https://github.com/apache/spark/commit/98e98b717786508fa76b8ddf0732a6672613e56a
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org