You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/07/07 10:28:12 UTC

[PR] [SPARK-57999][SQL][TESTS] Add generated table of accepted input types for built-in scalar functions [spark]

HyukjinKwon opened a new pull request, #57079:
URL: https://github.com/apache/spark/pull/57079

   ### What changes were proposed in this pull request?
   
   This adds `FunctionAcceptedTypesSuite` and a golden file recording, for every built-in
   **scalar** function and every argument position, which input `DataType`s the analyzer
   accepts, rejects, or cannot classify.
   
   For each function + argument position it probes every candidate type (all atomic types plus
   `NULL`, `CalendarInterval`, `Variant`, and representative complex types) by building
   `function(args…)` as an `UnresolvedFunction` with typed-`null` literals and running it through
   the analyzer. Each type is bucketed as:
   
   - **AcceptedDeclared** — accepted, and declared natively by the resolved expression via
     `ExpectsInputTypes` (empty means the expression validates ad hoc — *not* that it rejects
     everything).
   - **AcceptedViaCast** — all types the analyzer accepts, including via implicit cast.
   - **Rejected** — `DATATYPE_MISMATCH`.
   - **Inconclusive** — other analysis errors (e.g. a foldable-only or value constraint) that are
     not a pure type mismatch.
   
   Results are written to
   `sql/core/src/test/resources/sql-functions/sql-function-accepted-types.md`, regenerated with
   `SPARK_GENERATE_GOLDEN_FILES=1` (same convention as `ExpressionsSchemaSuite`). The current
   table covers 433 functions / 653 function-position rows. Functions with no analyzable baseline
   argument list (e.g. many aggregate/window/generator functions, or functions requiring specific
   foldable/enum arguments) are omitted for now and can be added in follow-ups.
   
   The suite is tagged `@ExtendedSQLTest`, so it is **skipped in the default test run** (it probes
   every function against every candidate type) and runs only in the dedicated "extended tests" CI
   leg — no build or CI configuration change is required.
   
   ### Why are the changes needed?
   
   Spark's built-in functions do not consistently document which input types each argument accepts.
   The authoritative constraint lives only on the catalyst expression (`inputTypes` /
   `checkInputDataTypes`), while `functions.scala` and `pyspark.sql.functions` are name-based
   facades over it. Following the principle that we should not document what is not tested, this
   suite establishes a **test-backed source of truth** for accepted types. Planned follow-ups use
   this golden table to add and verify accepted-type documentation in SQL `@ExpressionDescription`,
   Scala Scaladoc, and PySpark docstrings.
   
   The `AcceptedDeclared` vs `AcceptedViaCast` split is deliberate: e.g. `upper` declares only
   `STRING` but the analyzer accepts almost any atomic type via implicit cast; documenting the
   declared set is the honest answer, with the cast behaviour available alongside.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Test-only (adds a test suite and a test-resource golden file).
   
   ### How was this patch tested?
   
   ```
   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *FunctionAcceptedTypesSuite" -Dtest.include.tags=org.apache.spark.tags.ExtendedSQLTest
   ```
   then re-ran without regeneration to confirm the golden comparison passes deterministically, and
   confirmed the suite does not run in the default (non-extended) test configuration. Spot-checked
   the golden rows for `abs`, `upper`, `date_add`, `array_contains`.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Yes, drafted with assistance from Claude.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4932378691

   cc @cloud-fan too


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4932765525

   If you don't mind, shall we split this into more smaller reviewable PRs? For example, at least two; one for SQL, one for Python, @HyukjinKwon ? Although we live in agentic era, the PR should be human-reviewable as much as possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4932851959

   Yeah will do


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4932740704

   Could you check the linter failures?
   
   ```
   Scalastyle checks failed at following occurrences:
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:2967: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:2996: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:3864: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:3899: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:3934: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4190: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4212: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4937: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4958: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:4980: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:5001: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:5047: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:5093: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:8757: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:8884: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:8935: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:9355: File line length exceeds 100 characters
   [error] /__w/spark/spark/sql/api/src/main/scala/org/apache/spark/sql/functions.scala:9408: File line length exceeds 100 characters
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [DO-NOT-MERGE][SQL][PYTHON][DOCS] Document accepted input types for built-in functions, backed by a generated test [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4913469360

   **CI status** (fork runs on HyukjinKwon/spark)
   
   Last full run [`28921318633`](https://github.com/HyukjinKwon/spark/actions/runs/28921318633): **31/33 legs passed**, including all `sql - {other,slow,extended} tests` and every `pyspark-*` leg — so the accepted-types consistency tests (`FunctionAcceptedTypesSuite`, `FunctionScaladocAcceptedTypesSuite`, `ExpressionInfoSuite`/SPARK-57999, `test_function_accepted_types`) are green.
   
   Two legs failed:
   - **Scala linter** — two Scaladoc comment lines exceeded 100 chars; fixed in `cde5f2b80e7` (re-running now in [`28931137391`](https://github.com/HyukjinKwon/spark/actions/runs/28931137391)).
   - **Documentation generation** — pre-existing failure on `master`: `union` type-erasure clashes in `JavaSparkContext`/`JavaStreamingContext` and `StatefulProcessor.implicits$`, in files this PR does not touch (confirmed absent from the diff). Not introduced by this change.
   
   This is a [DO-NOT-MERGE] draft; posting for visibility.
   
   ---
   <!-- GITHUB_MCP_FOOTER: This attribution is automatically appended by GitHub MCP. -->
   _This comment was generated with [GitHub MCP](http://go/mcps)._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3548908643


##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -14992,6 +15356,7 @@ def decode(col: "ColumnOrName", charset: str) -> Column:
     -------
     :class:`~pyspark.sql.Column`
         the column for computed results.
+        Returns a column of the same type as the input.

Review Comment:
   `input` is a binary type.
   
   > Computes the first argument into a string from a binary using the provided character set



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3548918145


##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -692,8 +697,10 @@ def try_divide(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     ----------
     left : :class:`~pyspark.sql.Column` or column name
         dividend
+        A column that evaluates to a double or decimal.

Review Comment:
   Since there is no period at the previous description. Is it rendered like `dividend A column that evaluates to…` in the end?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3555617166


##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -4272,9 +4380,10 @@ object functions {
 
   /**
    * @param e
-   *   hyperbolic angle
+   *   hyperbolic angle. A column that evaluates to a double.
    * @return
-   *   hyperbolic sine of the given value, as if computed by `java.lang.Math.sinh`
+   *   hyperbolic sine of the given value, as if computed by `java.lang.Math.sinh` Returns a

Review Comment:
   The run-on sweep missed 17 `@return` blocks — here the appended sentence still fuses onto the prior one ("...`java.lang.Math.sinh` Returns a column...") without a separating period. The full remaining set (same fix — add a period before "Returns"): `sinh` (4385), `tanh` (4431), `add_months` (7831), `date_format` (7923), `year`/`quarter`/`month`/`dayofweek`/`dayofmonth`/`dayofyear` and siblings (8084, 8094, 8104, 8115, 8125, 8135, 8145, 8157), `last_day` (8238), and 8351/8365/8381. The other ~33 blocks were fixed correctly.
   ```suggestion
      *   hyperbolic sine of the given value, as if computed by `java.lang.Math.sinh`. Returns a
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4937637529

   Thank you for the decision, @HyukjinKwon !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3556084785


##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8986,6 +10385,8 @@ object functions {
    *
    * @group collection_funcs
    * @since 2.4.0
+   * @return
+   *   Returns a column of the element type of the input array.

Review Comment:
   `element_at` also accepts a map, where it returns the **value type** (`ElementAt.dataType`: `case MapType(_, valueType, _) => valueType`; the prose above says "Returns value for the given key ... if column is map"). The `@return` currently covers only the array case:
   ```suggestion
      *   Returns a column of the element type of the input array, or the value type of the input map.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9435,6 +10890,8 @@ object functions {
    *
    * @group generator_funcs
    * @since 1.3.0
+   * @return
+   *   Returns a column of the element type of the input array.

Review Comment:
   `explode` accepts a map too (`TypeCollection(ArrayType, MapType)`); for a map it produces `key` and `value` columns, not a single element type.
   ```suggestion
      *   Returns a column of the element type of the input array, or the key and value columns of the input map.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8999,6 +10400,8 @@ object functions {
    *
    * @group collection_funcs
    * @since 3.5.0
+   * @return
+   *   Returns a column of the element type of the input array.

Review Comment:
   Same as `element_at` (`try_element_at` wraps `ElementAt`, same `dataType`): also returns the map value type.
   ```suggestion
      *   Returns a column of the element type of the input array, or the value type of the input map.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19653,14 +20249,17 @@ def element_at(col: "ColumnOrName", extraction: Any) -> Column:
     Parameters
     ----------
     col : :class:`~pyspark.sql.Column` or str
-        name of column containing array or map
+        name of column containing array or map.
+        A column that evaluates to an array or map.
     extraction :
-        index to check for in array or key to check for in map
+        index to check for in array or key to check for in map.
+        A column that evaluates to an integer for an array, or the key type for a map.
 
     Returns
     -------
     :class:`~pyspark.sql.Column`
         value at given position.
+        Returns a column of the element type of the input array.

Review Comment:
   `element_at` also accepts a map, returning the value type (the docstring above says "Returns value for given key in `extraction` if col is map").
   ```suggestion
           Returns a column of the element type of the input array, or the value type of the input map.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9445,6 +10902,8 @@ object functions {
    *
    * @group generator_funcs
    * @since 2.2.0
+   * @return
+   *   Returns a column of the element type of the input array.

Review Comment:
   Same as `explode`: `explode_outer` on a map yields `key`/`value` columns.
   ```suggestion
      *   Returns a column of the element type of the input array, or the key and value columns of the input map.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -20779,6 +21414,7 @@ def explode(col: "ColumnOrName") -> Column:
     -------
     :class:`~pyspark.sql.Column`
         One row per array item or map key value.
+        Returns a column of the element type of the input array.

Review Comment:
   `explode` accepts a map too — the docstring says "Returns a new row for each element in the given array or map" and has "Example 2: Exploding a map column" (yielding `key`/`value`).
   ```suggestion
           Returns a column of the element type of the input array, or the key and value columns of the input map.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -21093,6 +21729,7 @@ def explode_outer(col: "ColumnOrName") -> Column:
     -------
     :class:`~pyspark.sql.Column`
         one row per array item or map key value.
+        Returns a column of the element type of the input array.

Review Comment:
   Same as `explode`: `explode_outer` on a map yields `key`/`value` columns.
   ```suggestion
           Returns a column of the element type of the input array, or the key and value columns of the input map.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19749,9 +20348,16 @@ def try_element_at(col: "ColumnOrName", extraction: "ColumnOrName") -> Column:
     Parameters
     ----------
     col : :class:`~pyspark.sql.Column` or str
-        name of column containing array or map
+        name of column containing array or map.
+        A column that evaluates to an array or map.
     extraction :
-        index to check for in array or key to check for in map
+        index to check for in array or key to check for in map.
+        A column that evaluates to an integer for an array, or the key type for a map.
+
+    Returns
+    -------
+    :class:`~pyspark.sql.Column`
+        Returns a column of the element type of the input array.

Review Comment:
   Same as `element_at`: `try_element_at` also returns the map value type.
   ```suggestion
           Returns a column of the element type of the input array, or the value type of the input map.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3550558552


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/First.scala:
##########
@@ -36,6 +36,12 @@ import org.apache.spark.sql.types._
   usage = """
     _FUNC_(expr[, isIgnoreNull]) - Returns the first value of `expr` for a group of rows.
       If `isIgnoreNull` is true, returns only non-null values.""",
+  arguments = """
+    Arguments:
+      * expr - The expression to return the first value of.
+        An expression of any type.
+      * isIgnoreNull - Whether to ignore null values and return only the first non-null value.

Review Comment:
   `isIgnoreNull` must be a boolean literal (`FirstLast.validateIgnoreNullExpr` accepts only `Literal(b: Boolean, BooleanType)`), so it warrants a type line like the other documented flags.
   ```suggestion
         * isIgnoreNull - Whether to ignore null values and return only the first non-null value.
           An expression that evaluates to a boolean. Must be a constant.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Last.scala:
##########
@@ -35,6 +35,12 @@ import org.apache.spark.sql.types._
   usage = """
     _FUNC_(expr[, isIgnoreNull]) - Returns the last value of `expr` for a group of rows.
       If `isIgnoreNull` is true, returns only non-null values""",
+  arguments = """
+    Arguments:
+      * expr - The expression to return the last value of.
+        An expression of any type.
+      * isIgnoreNull - Whether to ignore null values and return only the last non-null value.

Review Comment:
   Same as `first`: `isIgnoreNull` is a boolean constant.
   ```suggestion
         * isIgnoreNull - Whether to ignore null values and return only the last non-null value.
           An expression that evaluates to a boolean. Must be a constant.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/AnyValue.scala:
##########
@@ -34,6 +34,12 @@ import org.apache.spark.sql.types._
   usage = """
     _FUNC_(expr[, isIgnoreNull]) - Returns some value of `expr` for a group of rows.
       If `isIgnoreNull` is true, returns only non-null values.""",
+  arguments = """
+    Arguments:
+      * expr - The expression to return some value of for a group of rows.
+        An expression of any type.
+      * isIgnoreNull - Whether to return only non-null values.

Review Comment:
   Same: `isIgnoreNull` is a boolean constant.
   ```suggestion
         * isIgnoreNull - Whether to return only non-null values.
           An expression that evaluates to a boolean. Must be a constant.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Mode.scala:
##########
@@ -233,6 +233,13 @@ case class Mode(
     _FUNC_() WITHIN GROUP (ORDER BY col) - Returns the most frequent value for the values within `col` (specified in ORDER BY clause). NULL values are ignored.
       If all the values are NULL, or there are 0 rows, returns NULL. When multiple values have the same greatest frequency only one value will be returned.
       The value will be chosen based on sort direction. Return the smallest value if sort direction is asc or the largest value if sort direction is desc from multiple values with the same frequency.""",
+  arguments = """
+    Arguments:
+      * col - The column to compute the most frequent value of.
+        An expression of any type.
+      * deterministic - Whether to return a deterministic result when there are multiple most
+      frequent values.

Review Comment:
   `deterministic` is enforced foldable-boolean (Mode.scala:275-283). Adding the type line here; also fixes the continuation indent of the wrapped description.
   ```suggestion
         * deterministic - Whether to return a deterministic result when there are multiple most
             frequent values.
           An expression that evaluates to a boolean. Must be a constant.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala:
##########
@@ -1044,6 +1060,12 @@ case class MapSort(base: Expression)
       elements for double/float type. Null elements will be placed at the beginning of the returned
       array in ascending order or at the end of the returned array in descending order.
   """,
+  arguments = """
+    Arguments:
+      * array - The array to sort.
+        An expression that evaluates to an array.
+      * ascendingOrder - Whether to sort in ascending order; false sorts in descending order.

Review Comment:
   `ascendingOrder` is enforced foldable-boolean (collectionOperations.scala:1092-1099).
   ```suggestion
         * ascendingOrder - Whether to sort in ascending order; false sorts in descending order.
           An expression that evaluates to a boolean. Must be a constant.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3549855444


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala:
##########
@@ -177,6 +182,13 @@ object RaiseErrorExpressionBuilder extends ExpressionBuilder {
  */
 @ExpressionDescription(
   usage = "_FUNC_(expr [, message]) - Throws an exception if `expr` is not true.",
+  arguments = """
+    Arguments:
+      * expr - The expression that is expected to be true.
+        An expression that evaluates to a string.

Review Comment:
   `assert_true(expr [, message])` throws if `expr` is not true, so `expr` is the boolean condition — it desugars to `If(left, Literal(null), RaiseError(right))` (misc.scala:199), where `left` is `If`'s predicate.
   ```suggestion
           An expression that evaluates to a boolean.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala:
##########
@@ -177,6 +182,13 @@ object RaiseErrorExpressionBuilder extends ExpressionBuilder {
  */
 @ExpressionDescription(
   usage = "_FUNC_(expr [, message]) - Throws an exception if `expr` is not true.",
+  arguments = """
+    Arguments:
+      * expr - The expression that is expected to be true.
+        An expression that evaluates to a string.
+      * message - The message for the exception thrown when the expression is not true.
+        An expression that evaluates to a map.

Review Comment:
   `message` is the exception text (passed to `RaiseError`), a string — not a map. `raise_error` above documents its message correctly as "a string".
   ```suggestion
           An expression that evaluates to a string.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -14397,14 +14735,17 @@ def assert_true(col: "ColumnOrName", errMsg: Optional[Union[Column, str]] = None
     Parameters
     ----------
     col : :class:`~pyspark.sql.Column` or column name
-        column name or column that represents the input column to test
+        column name or column that represents the input column to test.
+        A column that evaluates to a string.

Review Comment:
   `col` is the boolean condition tested ("Returns `null` if the input column is `true`"), not a string.
   ```suggestion
           A column that evaluates to a boolean.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -14397,14 +14735,17 @@ def assert_true(col: "ColumnOrName", errMsg: Optional[Union[Column, str]] = None
     Parameters
     ----------
     col : :class:`~pyspark.sql.Column` or column name
-        column name or column that represents the input column to test
+        column name or column that represents the input column to test.
+        A column that evaluates to a string.
     errMsg : :class:`~pyspark.sql.Column` or literal string, optional
-        A Python string literal or column containing the error message
+        A Python string literal or column containing the error message.
+        A column that evaluates to a map.

Review Comment:
   `errMsg` is the error message string, not a map.
   ```suggestion
           A column that evaluates to a string.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -10212,6 +10608,8 @@ object functions {
    *
    * @group array_funcs
    * @since 2.4.0
+   * @return
+   *   Returns a column of the same type as the input.

Review Comment:
   `array_min` returns the array's **element** type, not the array type — `ArrayMin.dataType = ArrayType(dt,_) => dt`. "Same type as the input" is only correct for functions whose output equals their input type (`abs`, `reverse`, `round`, the `try_*` arithmetic, `nvl`, etc.), which are fine as-is.
   
   Same mislabel on the other six collection-element functions here (all say "same type as the input" but return the element/value type): `array_max` (10623), `element_at` (9351), `try_element_at` (9366), `get` (9378), `explode` (9824), `explode_outer` (9836) — and their PySpark mirrors. Suggest e.g. "Returns a column of the element type of the input array." (and for `element_at`/`try_element_at`, "...or the value type of the input map."; for `explode`, that it produces rows of the element type, or key/value columns for a map). This is the same issue @dongjoon-hyun flagged on `decode`.



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:
##########
@@ -308,6 +328,12 @@ trait CeilFloorExpressionBuilderBase extends ExpressionBuilder {
 // scalastyle:off line.size.limit
 @ExpressionDescription(
   usage = "_FUNC_(expr[, scale]) - Returns the smallest number after rounding up that is not smaller than `expr`. An optional `scale` parameter can be specified to control the rounding behavior.",
+  arguments = """
+    Arguments:
+      * expr - The expression to round up.
+        An expression that evaluates to a double, decimal, or long.
+      * scale - The number of decimal places to round to.

Review Comment:
   `scale` has no type and no constant marker, but `CeilFloorExpressionBuilderBase.build` requires `scale.foldable && scale.dataType == IntegerType`. The peer `round`/`bround` document the identical `d` argument fully (mathExpressions.scala:1885).
   ```suggestion
         * scale - The number of decimal places to round to.
           An expression that evaluates to an integer. Must be a constant.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala:
##########
@@ -2872,6 +2981,13 @@ case class ElementAt(
     _FUNC_(map, key) - Returns value for given key. The function always returns NULL
       if the key is not contained in the map.
   """,
+  arguments = """
+    Arguments:
+      * array - The array or map to retrieve an element from.
+        An expression that evaluates to an array.

Review Comment:
   The description line right above says "array or map", and `ElementAt` accepts `MapType` — so this drops the map case. For a map, `index` is the key (any type), not an integer. Compare `size`, which the PR documents as "an expression that evaluates to an array or map".
   ```suggestion
           An expression that evaluates to an array or map.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -19653,14 +20234,17 @@ def element_at(col: "ColumnOrName", extraction: Any) -> Column:
     Parameters
     ----------
     col : :class:`~pyspark.sql.Column` or str
-        name of column containing array or map
+        name of column containing array or map.
+        A column that evaluates to an array.

Review Comment:
   Same as the Scala side: the prose above says "array or map", and `element_at` accepts a map, so this should not be array-only. `try_element_at` and `get` have the same array-only param line. (Minor, non-blocking: PySpark `try_element_at` has no `Returns` block while `element_at` does.)
   ```suggestion
           A column that evaluates to an array or map.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:
##########
@@ -564,6 +617,12 @@ case class Floor(child: Expression) extends UnaryMathExpression(math.floor, "FLO
 // scalastyle:off line.size.limit
 @ExpressionDescription(
   usage = " _FUNC_(expr[, scale]) - Returns the largest number after rounding down that is not greater than `expr`. An optional `scale` parameter can be specified to control the rounding behavior.",
+  arguments = """
+    Arguments:
+      * expr - The expression to round down.
+        An expression that evaluates to a double, decimal, or long.
+      * scale - The number of decimal places to round to.

Review Comment:
   Same as `ceil`: `floor`'s `scale` must be a constant integer (`CeilFloorExpressionBuilderBase.build`), but the type and constant marker are missing.
   ```suggestion
         * scale - The number of decimal places to round to.
           An expression that evaluates to an integer. Must be a constant.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -22650,11 +23284,13 @@ def array_min(col: "ColumnOrName") -> Column:
     ----------
     col : :class:`~pyspark.sql.Column` or str
         The name of the column or an expression that represents the array.
+        A column that evaluates to an array.
 
     Returns
     -------
     :class:`~pyspark.sql.Column`
         A new column that contains the minimum value of each array.
+        Returns a column of the same type as the input.

Review Comment:
   Same as the Scala side: `array_min` returns the element type, not the array type. This "same type as the input" line is also wrong on `array_max`, `element_at`, `get`, `explode`, `explode_outer` in this file (all return the element/value type of a collection input).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3556885924


##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -20217,17 +20834,21 @@ def array_insert(arr: "ColumnOrName", pos: Union["ColumnOrName", int], value: An
     Parameters
     ----------
     arr : :class:`~pyspark.sql.Column` or str
-        name of column containing an array
+        name of column containing an array.
+        A column that evaluates to an array.
     pos : :class:`~pyspark.sql.Column` or str or int
         name of Numeric type column indicating position of insertion
-        (starting at index 1, negative position is a start from the back of the array)
+        (starting at index 1, negative position is a start from the back of the array).
+        A column of the same type as the array elements.

Review Comment:
   `pos` is the 1-based insertion index, not an array element — `ArrayInsert` coerces it to `IntegerType` (`inputTypes` returns `Seq(ArrayType(dt,_), IntegerType, dt)`, and `checkInputDataTypes` rejects a non-`IntegerType` `pos` with UNEXPECTED_INPUT_TYPE, collectionOperations.scala:5235-5257). The "element type" rewording was over-applied here; only the `value` arg below (20845) is the element type. The SQL `@ExpressionDescription` correctly keeps `pos` as integer (collectionOperations.scala:5208), so this makes PySpark disagree with SQL.
   ```suggestion
           A column that evaluates to an integer.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3550380252


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:
##########
@@ -593,6 +608,13 @@ case class BinaryPredicate(override val prettyName: String, left: Expression, ri
     Returns NULL if either input expression is NULL. Otherwise, returns False.
     Both left or right must be of STRING or BINARY type.
   """,
+  arguments = """
+    Arguments:
+      * left - The expression to search within.
+        An expression that evaluates to a string.

Review Comment:
   `contains` accepts binary as well as string — `StringBinaryPredicateExpressionBuilderBase.build` routes `(BinaryType, BinaryType)` to `BinaryPredicate`, and the usage above says "Both left or right must be of STRING or BINARY type". Same on `right` (616), and on `startswith` (663/665) and `endswith` (719/721) in this file.
   ```suggestion
           An expression that evaluates to a string or binary.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -18403,8 +18940,10 @@ def contains(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     ----------
     left : :class:`~pyspark.sql.Column` or str
         The input column or strings to check, may be NULL.
+        A column that evaluates to a string.

Review Comment:
   `contains` accepts binary too — the docstring says "Both left or right must be of STRING or BINARY type" and the example passes `to_binary(...)`. Same on `right` (18946), and on `startswith` (18629/18632) and `endswith` (18589/18592).
   ```suggestion
           A column that evaluates to a string or binary.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala:
##########
@@ -263,6 +267,11 @@ case class MinutesOfTime(child: Expression)
     sub-microsecond digits.
     If `expr` is a TIME type (since 4.1.0), it returns the minute of the time-of-day.
   """,
+  arguments = """
+    Arguments:
+      * expr - The expression to extract the minute component from.
+        An expression that evaluates to a timestamp.

Review Comment:
   `minute` accepts the TIME type as well as timestamp — the usage two lines up says "If `expr` is a TIME type (since 4.1.0), it returns the minute of the time-of-day", and `MinuteExpressionBuilder` dispatches `TimeType` to `MinutesOfTime` (`inputTypes = Seq(AnyTimeType)`). Same on `hour` (345) and `second` (442).
   ```suggestion
           An expression that evaluates to a timestamp or time.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -10408,6 +10626,7 @@ def minute(col: "ColumnOrName") -> Column:
     ----------
     col : :class:`~pyspark.sql.Column` or column name
         target date/time/timestamp column to work on.
+        A column that evaluates to a timestamp.

Review Comment:
   Same as the Scala side: `minute` accepts time as well as timestamp (the docstring notes "Added support for time type" and "target date/time/timestamp column"). Same on `hour` (10545) and `second` (10713).
   ```suggestion
           A column that evaluates to a timestamp or time.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:
##########
@@ -449,6 +456,12 @@ object JsonToStructs {
 // scalastyle:off line.size.limit
 @ExpressionDescription(
   usage = "_FUNC_(expr[, options]) - Returns a JSON string with a given struct value",
+  arguments = """
+    Arguments:
+      * expr - The struct value to convert to a JSON string.
+        An expression that evaluates to an array or struct.

Review Comment:
   `to_json` also accepts map and variant — `StructsToJson.checkInputDataTypes` accepts `StructType | MapType | ArrayType | VariantType` (map input is a common use). "array or struct" drops both.
   ```suggestion
           An expression that evaluates to a struct, array, map, or variant.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -21952,6 +22583,7 @@ def to_json(col: "ColumnOrName", options: Optional[Mapping[str, str]] = None) ->
     ----------
     col : :class:`~pyspark.sql.Column` or str
         name of column containing a struct, an array, a map, or a variant object.
+        A column that evaluates to an array or struct.

Review Comment:
   The docstring's first line already says "a StructType, ArrayType, MapType or a VariantType" and the param prose says "a struct, an array, a map, or a variant object" — so this type line should match rather than drop map and variant.
   ```suggestion
           A column that evaluates to a struct, array, map, or variant.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4932371728

   I think it's quite good now. I am triggering the last multi-agent orchastrated review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3551296811


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:
##########
@@ -223,6 +237,13 @@ case class TryToNumber(left: Expression, right: Expression)
         'hex': a string in the hexadecimal format.
         'utf-8': the input binary is decoded to UTF-8 string.
   """,
+  arguments = """
+    Arguments:
+      * expr - The expression to convert to a string.
+        An expression that evaluates to a decimal, timestamp, or time.

Review Comment:
   Still drops binary. `ToCharacterBuilder.build` routes a `BinaryType` input to `Base64`/`Hex`/`Decode`, and the examples below use `x'537061726b2053514c'` ('base64'/'hex') and `encode('abc', 'utf-8')`.
   ```suggestion
           An expression that evaluates to a decimal, timestamp, time, or binary.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -17310,8 +17814,10 @@ def to_char(col: "ColumnOrName", format: "ColumnOrName") -> Column:
     ----------
     col : :class:`~pyspark.sql.Column` or str
         Input column or strings.
+        A column that evaluates to a decimal, timestamp, or time.

Review Comment:
   Same as the Scala side: `to_char` accepts binary too (the docstring says "If `col` is a binary ...").
   ```suggestion
           A column that evaluates to a decimal, timestamp, time, or binary.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##########
@@ -58,6 +58,13 @@ case class TryEval(child: Expression) extends UnaryExpression {
 @ExpressionDescription(
   usage = "_FUNC_(expr1, expr2) - Returns the sum of `expr1`and `expr2` and the result is null on overflow. " +
     "The acceptable input types are the same with the `+` operator.",
+  arguments = """
+    Arguments:
+      * expr1 - The first addend.
+        An expression that evaluates to a numeric, interval, date, timestamp, or time.
+      * expr2 - The second addend.
+        An expression that evaluates to a numeric or interval.

Review Comment:
   `expr1` was widened but `expr2` wasn't. Addition is commutative — `BinaryArithmeticWithDatetimeResolver` handles `(interval, DateType)`, `(DayTimeInterval, TimeType)`, etc., so `expr2` accepts the same set as `expr1`.
   ```suggestion
           An expression that evaluates to a numeric, interval, date, timestamp, or time.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##########
@@ -219,6 +247,13 @@ case class TrySubtract(left: Expression, right: Expression, replacement: Express
 @ExpressionDescription(
   usage = "_FUNC_(expr1, expr2) - Returns `expr1`*`expr2` and the result is null on overflow. " +
     "The acceptable input types are the same with the `*` operator.",
+  arguments = """
+    Arguments:
+      * expr1 - The first factor.
+        An expression that evaluates to a numeric or interval.
+      * expr2 - The second factor.
+        An expression that evaluates to a numeric.

Review Comment:
   `expr2` should match `expr1` — `MultiplyInterval` handles both `(interval, numeric)` and `(numeric, interval)` (BinaryArithmeticWithDatetimeResolver.scala:149-150), so `numeric * interval` is valid.
   ```suggestion
           An expression that evaluates to a numeric or interval.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -553,7 +555,9 @@ def try_add(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     Parameters
     ----------
     left : :class:`~pyspark.sql.Column` or column name
+        A column that evaluates to a numeric, interval, date, timestamp, or time.
     right : :class:`~pyspark.sql.Column` or column name
+        A column that evaluates to a numeric or interval.

Review Comment:
   Same asymmetry as the Scala side: `try_add`'s second operand accepts the same types as the first (addition is commutative).
   ```suggestion
           A column that evaluates to a numeric, interval, date, timestamp, or time.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -17356,8 +17862,10 @@ def to_varchar(col: "ColumnOrName", format: "ColumnOrName") -> Column:
     ----------
     col : :class:`~pyspark.sql.Column` or str
         Input column or strings.
+        A column that evaluates to a decimal, timestamp, or time.

Review Comment:
   Same: `to_varchar` accepts binary.
   ```suggestion
           A column that evaluates to a decimal, timestamp, time, or binary.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -803,9 +812,11 @@ def try_multiply(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     Parameters
     ----------
     left : :class:`~pyspark.sql.Column` or column name
-        multiplicand
+        multiplicand.
+        A column that evaluates to a numeric or interval.
     right : :class:`~pyspark.sql.Column` or column name
-        multiplier
+        multiplier.
+        A column that evaluates to a numeric.

Review Comment:
   Same asymmetry: `try_multiply`'s second operand also accepts an interval.
   ```suggestion
           A column that evaluates to a numeric or interval.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3549259708


##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -692,8 +697,10 @@ def try_divide(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     ----------
     left : :class:`~pyspark.sql.Column` or column name
         dividend
+        A column that evaluates to a double or decimal.

Review Comment:
   let me take a look again ..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3556948033


##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -9983,7 +11506,8 @@ object functions {
    *   "https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option"> Data
    *   Source Option</a> in the version you use.
    * @return
-   *   a column with string literal containing schema in DDL format.
+   *   a column with string literal containing schema in DDL format. Returns a column that
+   *   evaluates to an integer.

Review Comment:
   This `schema_of_json` change looks wrong to me because we don't return `integer`?
   > Returns a column that evaluates to an integer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #57079:
URL: https://github.com/apache/spark/pull/57079#issuecomment-4933176574

   Superseded by smaller, per-surface PRs for reviewability (per @dongjoon-hyun's suggestion). SPARK-57999 is now an umbrella, with one sub-task and PR per documentation surface:
   
   - **SQL** (`@ExpressionDescription` reference): SPARK-58085 — #57183
   - **Scala** (`functions.scala`): SPARK-58086 — #57184
   - **PySpark** (`builtin.py`): SPARK-58087 — #57185
   
   Each carries the exact reviewed content from this PR for its surface (scalastyle line-length fixes and the `schema_of_json` return-type fix included). Closing this in favor of the three above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #57079: [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions
URL: https://github.com/apache/spark/pull/57079


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3548895328


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala:
##########
@@ -331,6 +331,11 @@ case class Percentile(
 
 @ExpressionDescription(
   usage = "_FUNC_(col) - Returns the median of numeric, ANSI interval or TIME column `col`.",
+  arguments = """
+    Arguments:
+      * col - The column to compute the median of.
+        An expression that evaluates to an interval or time.

Review Comment:
   `numeric` is missing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3550697174


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:
##########
@@ -223,6 +237,13 @@ case class TryToNumber(left: Expression, right: Expression)
         'hex': a string in the hexadecimal format.
         'utf-8': the input binary is decoded to UTF-8 string.
   """,
+  arguments = """
+    Arguments:
+      * expr - The expression to convert to a string.
+        An expression that evaluates to a numeric, datetime, or binary.

Review Comment:
   `to_char`/`to_varchar` accept datetime and binary as well as numeric — `ToCharacterBuilder.build` dispatches `DatetimeType` to `DateFormatClass` and `BinaryType` to `Base64`/`Hex`/`Decode`. The examples below use `date'2016-04-08'` and `x'537061726b2053514c'`, so "decimal" is too narrow.
   ```suggestion
           An expression that evaluates to a numeric, datetime, or binary.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -17310,8 +17799,10 @@ def to_char(col: "ColumnOrName", format: "ColumnOrName") -> Column:
     ----------
     col : :class:`~pyspark.sql.Column` or str
         Input column or strings.
+        A column that evaluates to a numeric, datetime, or binary.

Review Comment:
   Same as the Scala side: `to_char` accepts numeric, datetime, or binary (the docstring itself says "If `col` is a datetime ..." and "If `col` is a binary ...").
   ```suggestion
           A column that evaluates to a numeric, datetime, or binary.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -17356,8 +17847,10 @@ def to_varchar(col: "ColumnOrName", format: "ColumnOrName") -> Column:
     ----------
     col : :class:`~pyspark.sql.Column` or str
         Input column or strings.
+        A column that evaluates to a numeric, datetime, or binary.

Review Comment:
   Same as `to_char`: `to_varchar` accepts numeric, datetime, or binary.
   ```suggestion
           A column that evaluates to a numeric, datetime, or binary.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##########
@@ -58,6 +58,13 @@ case class TryEval(child: Expression) extends UnaryExpression {
 @ExpressionDescription(
   usage = "_FUNC_(expr1, expr2) - Returns the sum of `expr1`and `expr2` and the result is null on overflow. " +
     "The acceptable input types are the same with the `+` operator.",
+  arguments = """
+    Arguments:
+      * expr1 - The first addend.
+        An expression that evaluates to a numeric or interval.

Review Comment:
   `try_add` also accepts date/timestamp operands (date arithmetic) — the examples below use `try_add(date'2021-01-01', 1)` and `try_add(date'2021-01-01', interval 1 year)`. Non-blocking, but "numeric or interval" omits the date/timestamp case the examples show. Same on `expr2` (66).
   ```suggestion
           An expression that evaluates to a numeric, date, timestamp, or interval.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:
##########
@@ -177,6 +198,13 @@ case class TryMod(left: Expression, right: Expression, replacement: Expression)
 @ExpressionDescription(
   usage = "_FUNC_(expr1, expr2) - Returns `expr1`-`expr2` and the result is null on overflow. " +
     "The acceptable input types are the same with the `-` operator.",
+  arguments = """
+    Arguments:
+      * expr1 - The minuend to subtract from.
+        An expression that evaluates to a numeric or interval.

Review Comment:
   Same as `try_add`: `try_subtract` accepts date/timestamp operands too. Same on `expr2` (206).
   ```suggestion
           An expression that evaluates to a numeric, date, timestamp, or interval.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -867,7 +878,9 @@ def try_subtract(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     Parameters
     ----------
     left : :class:`~pyspark.sql.Column` or column name
+        A column that evaluates to a numeric or interval.

Review Comment:
   Same: `try_subtract`'s example uses a date operand.
   ```suggestion
           A column that evaluates to a numeric, date, timestamp, or interval.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -553,7 +555,9 @@ def try_add(left: "ColumnOrName", right: "ColumnOrName") -> Column:
     Parameters
     ----------
     left : :class:`~pyspark.sql.Column` or column name
+        A column that evaluates to a numeric or interval.

Review Comment:
   Same as the Scala side: the example uses `try_add("date", sf.lit(1))`, so date/timestamp operands should be reflected here. Non-blocking.
   ```suggestion
           A column that evaluates to a numeric, date, timestamp, or interval.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3555438612


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala:
##########
@@ -59,6 +59,15 @@ import org.apache.spark.unsafe.types.UTF8String
       `k` An optional INTEGER literal greater than 0. If k is not specified, it defaults to 5.
       `maxItemsTracked` An optional INTEGER literal greater than or equal to k and has upper limit of 1000000. If maxItemsTracked is not specified, it defaults to 10000.
   """,
+  arguments = """
+    Arguments:
+      * expr - The expression to compute the top k most frequent items of.
+        An expression of any type.

Review Comment:
   `expr` isn't actually "any type": `checkInputDataTypes` routes it through `ApproxTopK.isDataTypeSupported` (line 237-244), which accepts only boolean/byte/short/int/long/float/double/date/timestamp/timestamp_ntz/string/decimal — binary is explicitly excluded ("ItemsSketch seems cannot count the frequency correctly") and array/map/struct/interval/variant are omitted, so e.g. `approx_top_k(binary_col)` fails with `"binary columns are not supported"`. Unlike `mode`/`first`/`any_value` (genuine `AnyDataType`), this warrants the accepted atomic set.
   ```suggestion
           An expression that evaluates to a boolean, numeric, date, timestamp, or string.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala:
##########
@@ -488,6 +497,13 @@ object ApproxTopKAggregateBuffer {
     _FUNC_(expr, maxItemsTracked) - Accumulates items into a sketch.
       `maxItemsTracked` An optional positive INTEGER literal with upper limit of 1000000. If maxItemsTracked is not specified, it defaults to 10000.
   """,
+  arguments = """
+    Arguments:
+      * expr - The expression whose values are accumulated into the sketch.
+        An expression of any type.

Review Comment:
   Same as `approx_top_k` above: `approx_top_k_accumulate`'s `expr` is gated by `isDataTypeSupported`, not truly "any type".
   ```suggestion
           An expression that evaluates to a boolean, numeric, date, timestamp, or string.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -3314,12 +3354,15 @@ object functions {
    *
    * @group math_funcs
    * @since 1.3.0
+   * @return
+   *   Returns a column of the same type as the input.
    */
   def abs(e: Column): Column = Column.fn("abs", e)
 
   /**
    * @return
-   *   inverse cosine of `e` in radians, as if computed by `java.lang.Math.acos`
+   *   inverse cosine of `e` in radians, as if computed by `java.lang.Math.acos` Returns a column

Review Comment:
   The appended "Returns a column..." sentence has no separating period, so Scaladoc renders it as a run-on: "...as if computed by `java.lang.Math.acos` Returns a column that evaluates to a double." This is the same defect @dongjoon-hyun flagged on PySpark `builtin.py:700` (`dividend`) — fixed there in this revision, but the Scala `@return` blocks weren't swept. It affects ~50 `@return` blocks here (acos/acosh/asin/atan/atan2/cos/cot/sec/sin/tan family, `width_bucket`, `levenshtein`:5460, and the ~30 date-part extractors), while the `@param` appended sentences are all correctly punctuated. Add a period before "Returns" in each:
   ```suggestion
      *   inverse cosine of `e` in radians, as if computed by `java.lang.Math.acos`. Returns a column
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3555694226


##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8127,9 +8496,11 @@ object functions {
    *
    * @param s
    *   A date, timestamp or string. If a string, the data must be in a format that can be cast to
-   *   a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`
+   *   a timestamp, such as `yyyy-MM-dd` or `yyyy-MM-dd HH:mm:ss.SSSS`. A column that evaluates to
+   *   a string, date, timestamp, timestamp_ntz, or numeric.
    * @return
    *   A timestamp, or null if the input was a string that could not be cast to a timestamp

Review Comment:
   The sole `@return` run-on left after the 16-block fix: this line ends without a period, so it fuses with "Returns a column that evaluates to a timestamp." on the next line. My earlier comment said "17 blocks" but only enumerated 16 line sites, and `to_timestamp` wasn't among them — so it slipped through. Same fix as the rest:
   ```suggestion
      *   A timestamp, or null if the input was a string that could not be cast to a timestamp.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/AnyValue.scala:
##########
@@ -34,6 +34,13 @@ import org.apache.spark.sql.types._
   usage = """
     _FUNC_(expr[, isIgnoreNull]) - Returns some value of `expr` for a group of rows.
       If `isIgnoreNull` is true, returns only non-null values.""",
+  arguments = """
+    Arguments:
+      * expr - The expression to return some value of for a group of rows.

Review Comment:
   "...some value of **for** a group of rows." reads broken — the dangling "of" collides with the "for a group of rows" clause. The peers `first`/`last` use "The expression to return the {first,last} value of." — suggest matching them:
   ```suggestion
         * expr - The expression to return some value of.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57999][SQL][PYTHON][DOCS] Document accepted input and return types for built-in functions [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #57079:
URL: https://github.com/apache/spark/pull/57079#discussion_r3555785692


##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8160,7 +8531,7 @@ object functions {
    * @param str
    *   A string to be parsed to time.
    * @return
-   *   A time, or null if the input is malformed.
+   *   A time, or null if the input is malformed. Returns a column that evaluates to a string.

Review Comment:
   Same as `to_time`: `try_to_time` returns `TimeType`, not string.
   ```suggestion
      *   A time, or null if the input is malformed. Returns a column that evaluates to a time.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -11914,13 +12204,16 @@ def to_time(str: "ColumnOrName", format: Optional["ColumnOrName"] = None) -> Col
     ----------
     str : :class:`~pyspark.sql.Column` or column name
         string to be parsed to time.
+        A column that evaluates to a string.
     format: :class:`~pyspark.sql.Column` or column name, optional
         time format pattern to follow.
+        A column that evaluates to a string.
 
     Returns
     -------
     :class:`~pyspark.sql.Column`
         time value as :class:`pyspark.sql.types.TimeType` type.
+        Returns a column that evaluates to a string.

Review Comment:
   Wrong return type — mirrors the Scala `to_time`: this line sits directly under "time value as `pyspark.sql.types.TimeType` type.", so "string" contradicts it.
   ```suggestion
           Returns a column that evaluates to a time.
   ```



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala:
##########
@@ -78,6 +85,13 @@ case class BitwiseAnd(left: Expression, right: Expression) extends BinaryArithme
  */
 @ExpressionDescription(
   usage = "expr1 _FUNC_ expr2 - Returns the result of bitwise OR of `expr1` and `expr2`.",
+  arguments = """
+    Arguments:
+      * expr1 - The expr1 argument.
+        An expression that evaluates to an integral.
+      * expr2 - The expr2 argument.

Review Comment:
   These arg descriptions just restate the parameter name ("The expr1 argument.") — no information. The sibling `BitwiseAnd`/`BitwiseXor` in this file use "The first/second operand of the bitwise AND / exclusive OR."; suggest matching them:
   ```suggestion
         * expr1 - The first operand of the bitwise OR.
           An expression that evaluates to an integral.
         * expr2 - The second operand of the bitwise OR.
   ```



##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -12056,13 +12351,16 @@ def try_to_time(str: "ColumnOrName", format: Optional["ColumnOrName"] = None) ->
     ----------
     str : :class:`~pyspark.sql.Column` or column name
         string to be parsed to time.
+        A column that evaluates to a string.
     format: :class:`~pyspark.sql.Column` or column name, optional
         time format pattern to follow.
+        A column that evaluates to a string.
 
     Returns
     -------
     :class:`~pyspark.sql.Column`
         time value as :class:`pyspark.sql.types.TimeType` type.
+        Returns a column that evaluates to a string.

Review Comment:
   Same as `to_time`: `try_to_time` returns `TimeType`, not string.
   ```suggestion
           Returns a column that evaluates to a time.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -8093,7 +8461,8 @@ object functions {
    * @param str
    *   A string to be parsed to time.
    * @return
-   *   A time, or raises an error if the input is malformed.
+   *   A time, or raises an error if the input is malformed. Returns a column that evaluates to a
+   *   string.

Review Comment:
   Wrong return type: `to_time` returns `TimeType` (`ToTime.dataType = TimeType()`), not string — the prose right here even says "A time". Same on `try_to_time` (8534) and the PySpark mirrors (builtin.py:12216, 12363). This has been here since the first reviewed revision, so it's my own late catch, not from a recent commit.
   ```suggestion
      *   A time, or raises an error if the input is malformed. Returns a column that evaluates to a
      *   time.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org