You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2026/08/04 11:14:33 UTC
(spark) branch master updated: [SPARK-58359][SQL] Require the arguments field in built-in function documentation
This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 5c7867479c66 [SPARK-58359][SQL] Require the arguments field in built-in function documentation
5c7867479c66 is described below
commit 5c7867479c66a16b7f5ac0e940262b9f7a990692
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 4 20:14:18 2026 +0900
[SPARK-58359][SQL] Require the arguments field in built-in function documentation
### What changes were proposed in this pull request?
This PR makes the `arguments` field of `ExpressionDescription` a required part of built-in function documentation, alongside the already-required `usage`, `examples`, `since`, and `group` fields.
- `ExpressionInfoSuite` (the `SPARK-32870` test) now asserts `info.getArguments.nonEmpty` for every registered function, except functions that take no arguments, which are enumerated in a new `noArgumentsSet` (e.g. `pi`, `current_date`, `current_timestamp`, `input_file_name`, `uuid`, `spark_partition_id`). The existing format assertions (the value must start with `\n Arguments:\n` and end with `\n `) continue to apply.
- Backfills `arguments` documentation for 118 built-in expressions across 45 files that were previously missing it.
Note on scope: the enforced list is derived from what is actually registered in the expression `FunctionRegistry`. Expressions whose `ExpressionInfo` is produced through a builder object are validated via their builder class; some of these already document `arguments` (e.g. `minute`/`second`) and were not part of the backfill, while others whose builders were missing it (e.g. `max_by`/`min_by`) were backfilled on the builder class. Expressions that live only in the table function regi [...]
### Why are the changes needed?
Most built-in functions already document their `arguments`, but it was not enforced, so newly added expressions could omit it and the generated function docs would be inconsistent. Requiring the field keeps the built-in function documentation complete and uniform, and turns a missing `Arguments:` section into a test failure rather than a silent gap.
### Does this PR introduce _any_ user-facing change?
Yes. The generated documentation for the affected built-in functions now includes an `Arguments:` section describing each argument. This is a documentation-only change; there are no behavior changes.
### How was this patch tested?
Existing `ExpressionInfoSuite` covers this, extended to assert that `arguments` is present for all registered functions (minus the zero-argument exemptions).
```
build/sbt 'sql/testOnly org.apache.spark.sql.expressions.ExpressionInfoSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
This pull request and its description were written by Isaac.
Closes #57551 from HyukjinKwon/docs-require-arguments.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <hy...@databricks.com>
---
.../expressions/ApproxTopKExpressions.scala | 7 ++++
.../expressions/CallMethodViaReflection.scala | 7 ++++
.../spark/sql/catalyst/expressions/Cast.scala | 5 +++
.../spark/sql/catalyst/expressions/TryEval.scala | 6 +++
.../aggregate/ApproxTopKAggregates.scala | 9 +++++
.../aggregate/ApproximatePercentile.scala | 10 +++++
.../sql/catalyst/expressions/aggregate/Count.scala | 5 +++
.../expressions/aggregate/CountMinSketchAgg.scala | 7 ++++
.../expressions/aggregate/HistogramNumeric.scala | 6 +++
.../aggregate/HyperLogLogPlusPlus.scala | 6 +++
.../sql/catalyst/expressions/aggregate/Max.scala | 5 +++
.../catalyst/expressions/aggregate/MaxMinByK.scala | 14 +++++++
.../sql/catalyst/expressions/aggregate/Min.scala | 5 +++
.../catalyst/expressions/aggregate/collect.scala | 12 ++++++
.../aggregate/datasketchesAggregates.scala | 8 ++++
.../expressions/aggregate/kllAggregates.scala | 46 +++++++++++++++++++---
.../expressions/aggregate/percentiles.scala | 21 ++++++++++
.../aggregate/thetasketchesAggregates.scala | 11 ++++++
.../sql/catalyst/expressions/arithmetic.scala | 19 +++++++++
.../catalyst/expressions/avroSqlFunctions.scala | 18 +++++++++
.../catalyst/expressions/bitmapExpressions.scala | 15 +++++++
.../expressions/collectionOperations.scala | 42 ++++++++++++++++++++
.../catalyst/expressions/complexTypeCreator.scala | 27 ++++++++++++-
.../expressions/conditionalExpressions.scala | 6 +++
.../sql/catalyst/expressions/csvExpressions.scala | 21 ++++++++++
.../expressions/datasketchesExpressions.scala | 14 +++++++
.../catalyst/expressions/datetimeExpressions.scala | 4 ++
.../sql/catalyst/expressions/generators.scala | 21 ++++++++++
.../spark/sql/catalyst/expressions/grouping.scala | 11 ++++++
.../spark/sql/catalyst/expressions/hash.scala | 10 +++++
.../expressions/higherOrderFunctions.scala | 26 ++++++++++++
.../sql/catalyst/expressions/jsonExpressions.scala | 20 ++++++++++
.../spark/sql/catalyst/expressions/misc.scala | 4 ++
.../sql/catalyst/expressions/nullExpressions.scala | 28 +++++++++++++
.../catalyst/expressions/stringExpressions.scala | 6 +++
.../expressions/thetasketchesExpressions.scala | 29 ++++++++++++++
.../expressions/toFromProtobufSqlFunctions.scala | 23 +++++++++++
.../sql/catalyst/expressions/tupleDifference.scala | 22 +++++++++++
.../catalyst/expressions/tupleIntersection.scala | 34 ++++++++++++++++
.../catalyst/expressions/tupleSketchEstimate.scala | 20 ++++++++++
.../catalyst/expressions/tupleSketchSummary.scala | 14 +++++++
.../sql/catalyst/expressions/tupleUnion.scala | 36 +++++++++++++++++
.../expressions/variant/variantExpressions.scala | 35 ++++++++++++++++
.../catalyst/expressions/vectorExpressions.scala | 41 +++++++++++++++++++
.../sql/catalyst/expressions/xmlExpressions.scala | 17 ++++++++
.../sql/expressions/ExpressionInfoSuite.scala | 33 +++++++++++++++-
46 files changed, 778 insertions(+), 8 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
index 8d4f0a18d570..fc647c860208 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ApproxTopKExpressions.scala
@@ -42,6 +42,13 @@ import org.apache.spark.sql.types._
_FUNC_(state, k) - Returns top k items with their frequency.
`k` An optional INTEGER literal greater than 0. If k is not specified, it defaults to 5.
""",
+ arguments = """
+ Arguments:
+ * state - The sketch state produced by `approx_top_k_accumulate` or
+ `approx_top_k_combine`.
+ * k - Optional. A constant INTEGER literal greater than 0 giving the number
+ of top items to return. If omitted, it defaults to 5.
+ """,
examples = """
Examples:
> SELECT _FUNC_(approx_top_k_accumulate(expr)) FROM VALUES (0), (0), (1), (1), (2), (3), (4), (4) AS tab(expr);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
index 90b9f19cf8d0..841648912df8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CallMethodViaReflection.scala
@@ -51,6 +51,13 @@ import org.apache.spark.util.Utils
*/
@ExpressionDescription(
usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - Calls a method with reflection.",
+ arguments = """
+ Arguments:
+ * class - A literal string with the fully qualified name of the class.
+ * method - A literal string with the name of the static method to call.
+ * argN - Optional arguments passed to the method. Only primitive and string
+ types are supported, and each argument is matched to the method signature.
+ """,
examples = """
Examples:
> SELECT _FUNC_('java.util.UUID', 'randomUUID');
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 72979889a3ba..b1fcee354b83 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -660,6 +660,11 @@ object Cast extends QueryErrorsBase {
@ExpressionDescription(
usage = "_FUNC_(expr AS type) - Casts the value `expr` to the target data type `type`." +
" `expr` :: `type` alternative casting syntax is also supported.",
+ arguments = """
+ Arguments:
+ * expr - An expression whose value is converted to the target data type.
+ * type - The target data type to cast the value to.
+ """,
examples = """
Examples:
> SELECT _FUNC_('10' as int);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
index 289b102c1f6f..b41f690f8690 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala
@@ -329,6 +329,12 @@ case class TryToBinary(
@ExpressionDescription(
usage = "_FUNC_(class, method[, arg1[, arg2 ..]]) - This is a special version of `reflect` that" +
" performs the same operation, but returns a NULL value instead of raising an error if the invoke method thrown exception.",
+ arguments = """
+ Arguments:
+ * class - A string literal with the fully qualified name of the class.
+ * method - A string literal with the name of the static method to invoke.
+ * arg1, arg2, ... - Optional arguments passed to the invoked method.
+ """,
examples = """
Examples:
> SELECT _FUNC_('java.util.UUID', 'randomUUID');
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
index f9023b683e11..8ed6590793da 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala
@@ -798,6 +798,15 @@ object CombineInternal {
_FUNC_(state, maxItemsTracked) - Combines multiple sketches into a single sketch.
`maxItemsTracked` An optional positive INTEGER literal with upper limit of 1000000. If maxItemsTracked is specified, it will be set for the combined sketch. If maxItemsTracked is not specified, the input sketches must have the same maxItemsTracked value, otherwise an error will be thrown. The output sketch will use the same value from the input sketches.
""",
+ arguments = """
+ Arguments:
+ * state - The sketch state to combine, as produced by approx_top_k_accumulate.
+ An expression that evaluates to the sketch state struct.
+ * maxItemsTracked - Optional. The maximum number of items to track in the combined
+ sketch, with an upper limit of 1000000. An expression that evaluates to an integer.
+ Must be a constant. If not specified, the input sketches must share the same
+ maxItemsTracked value, which is used for the output sketch.
+ """,
examples = """
Examples:
> SELECT approx_top_k_estimate(_FUNC_(sketch, 10000), 5) FROM (SELECT approx_top_k_accumulate(expr) AS sketch FROM VALUES (0), (0), (1), (1) AS tab(expr) UNION ALL SELECT approx_top_k_accumulate(expr) AS sketch FROM VALUES (2), (3), (4), (4) AS tab(expr));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
index 19e30c53b436..b9caae5467ce 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
@@ -66,6 +66,16 @@ import org.apache.spark.util.ArrayImplicits._
In this case, returns the approximate percentile array of column `col` at the given
percentage array.
""",
+ arguments = """
+ Arguments:
+ * col - The numeric, ANSI interval or TIME column whose percentile is
+ computed.
+ * percentage - A value (or array of values) between 0.0 and 1.0 specifying
+ the percentile(s) to compute.
+ * accuracy - Optional. A positive numeric literal (default: 10000) that
+ controls approximation accuracy at the cost of memory. Higher values
+ yield better accuracy; `1.0/accuracy` is the relative error.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col, array(0.5, 0.4, 0.1), 100) FROM VALUES (0), (1), (2), (10) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
index 758ef22f0a2c..f5f248c48387 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Count.scala
@@ -34,6 +34,11 @@ import org.apache.spark.sql.types._
_FUNC_(DISTINCT expr[, expr...]) - Returns the number of rows for which the supplied expression(s) are unique and non-null.
""",
+ arguments = """
+ Arguments:
+ * expr - One or more expressions. A row is counted only when all supplied
+ expressions are non-null. Use `*` to count all rows, including rows with nulls.
+ """,
examples = """
Examples:
> SELECT _FUNC_(*) FROM VALUES (NULL), (5), (5), (20) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
index f0a27677628d..a1eb024488bd 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountMinSketchAgg.scala
@@ -206,6 +206,13 @@ case class CountMinSketchAgg(
`CountMinSketch` before usage. Count-min sketch is a probabilistic data structure used for
cardinality estimation using sub-linear space.
""",
+ arguments = """
+ Arguments:
+ * col - The column to build the count-min sketch from.
+ * eps - A double literal for the relative error of the sketch.
+ * confidence - A double literal for the confidence of the sketch.
+ * seed - An integer literal used as the random seed.
+ """,
examples = """
Examples:
> SELECT hex(_FUNC_(col, 0.5d, 0.5d, 1)) FROM VALUES (1), (2), (1) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
index dadf7ac53c59..fe89a535f264 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HistogramNumeric.scala
@@ -51,6 +51,12 @@ import org.apache.spark.sql.util.NumericHistogram
statistical computing packages. Note: the output type of the 'x' field in the return value is
propagated from the input value consumed in the aggregate function.
""",
+ arguments = """
+ Arguments:
+ * expr - A numeric, date, timestamp, or interval expression whose values are aggregated
+ into the histogram.
+ * nb - A foldable integer expression (at least 2) giving the number of histogram bins.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col, 5) FROM VALUES (0), (1), (2), (10) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
index f304b43358ad..56f203aa9c4e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
@@ -50,6 +50,12 @@ import org.apache.spark.sql.types._
usage = """
_FUNC_(expr[, relativeSD]) - Returns the estimated cardinality by HyperLogLog++.
`relativeSD` defines the maximum relative standard deviation allowed.""",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose distinct values are counted.
+ * relativeSD - An optional double literal for the maximum relative standard
+ deviation allowed. Defaults to 0.05.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col1) FROM VALUES (1), (1), (2), (2), (3) tab(col1);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
index f49297eba88b..f2a36e571cd8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Max.scala
@@ -27,6 +27,11 @@ import org.apache.spark.sql.types._
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the maximum value of `expr`.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type whose maximum value across the group is
+ returned. NULL values are ignored.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (10), (50), (20) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
index a0a7acd93097..567beb9a7484 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/MaxMinByK.scala
@@ -240,6 +240,13 @@ case class MaxMinByK(
maximum values of `y`, sorted in descending order by `y`.
Returns NULL if there are no non-NULL ordering values.
""",
+ arguments = """
+ Arguments:
+ * x - The value expression to return.
+ * y - The ordering expression whose maximum selects the value of `x`.
+ * k - An optional positive integer. When present, returns an array of the `k` values
+ of `x` associated with the largest values of `y`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(x, y) FROM VALUES ('a', 10), ('b', 50), ('c', 20) AS tab(x, y);
@@ -275,6 +282,13 @@ object MaxByBuilder extends ExpressionBuilder {
minimum values of `y`, sorted in ascending order by `y`.
Returns NULL if there are no non-NULL ordering values.
""",
+ arguments = """
+ Arguments:
+ * x - The value expression to return.
+ * y - The ordering expression whose minimum selects the value of `x`.
+ * k - An optional positive integer. When present, returns an array of the `k` values
+ of `x` associated with the smallest values of `y`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(x, y) FROM VALUES ('a', 10), ('b', 50), ('c', 20) AS tab(x, y);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
index eaef7b6bec11..7585b1e80b2c 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Min.scala
@@ -27,6 +27,11 @@ import org.apache.spark.sql.types._
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns the minimum value of `expr`.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type whose minimum value across the group is
+ returned. NULL values are ignored.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (10), (-1), (20) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
index c3385bc39026..0ed76e0bfa7d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala
@@ -104,6 +104,10 @@ abstract class Collect[T <: Growable[Any] with Iterable[Any]] extends TypedImper
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Collects and returns a list of non-unique elements.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose values are collected into a list.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (1), (2), (1) AS tab(col);
@@ -181,6 +185,10 @@ case class CollectList(
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Collects and returns a set of unique elements.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose values are collected into a set.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (1), (2), (1) AS tab(col);
@@ -336,6 +344,10 @@ case class CollectSet(
@ExpressionDescription(
usage =
"_FUNC_(expr) - Collects and returns the distinct union of the elements of array `expr`.",
+ arguments = """
+ Arguments:
+ * expr - An array expression whose elements are collected into a set across rows.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (array(1, 2)), (array(2, 3)), (array(1)) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
index 952b331b7227..4470c415f5c9 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala
@@ -243,6 +243,14 @@ object HllSketchAgg {
_FUNC_(expr, allowDifferentLgConfigK) - Returns the merged HllSketch's updatable binary representation.
`allowDifferentLgConfigK` (optional) Allow sketches with different lgConfigK values
to be unioned (defaults to false).""",
+ arguments = """
+ Arguments:
+ * expr - The binary representation of an HllSketch to merge.
+ An expression that evaluates to binary.
+ * allowDifferentLgConfigK - Optional. Whether to allow sketches with different
+ lgConfigK values to be unioned. An expression that evaluates to a boolean.
+ Defaults to false.
+ """,
examples = """
Examples:
> SELECT hll_sketch_estimate(_FUNC_(sketch, true)) FROM (SELECT hll_sketch_agg(col) as sketch FROM VALUES (1) tab(col) UNION ALL SELECT hll_sketch_agg(col, 20) as sketch FROM VALUES (1) tab(col));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
index a811736c5ee2..f13bad8291c6 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala
@@ -57,9 +57,10 @@ import org.apache.spark.sql.types.{AbstractDataType, BinaryType, ByteType, DataT
arguments = """
Arguments:
* expr - The expression to aggregate into the KLL sketch.
- An expression that evaluates to an integral.
- * k - The parameter controlling the size and accuracy of the sketch.
- An expression that evaluates to an integer. Must be a constant.
+ An expression that evaluates to an integral.
+ * k - Optional. The parameter controlling the size and accuracy of the sketch.
+ An expression that evaluates to an integer between 8 and 65535. Must be a
+ constant. Defaults to 200.
""",
examples = """
Examples:
@@ -208,6 +209,14 @@ case class KllSketchAggBigint(
The optional k parameter controls the size and accuracy of the sketch (default 200, range 8-65535).
Larger k values provide more accurate quantile estimates but result in larger, slower sketches.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to aggregate into the KLL sketch.
+ An expression that evaluates to a float.
+ * k - Optional. The parameter controlling the size and accuracy of the sketch.
+ An expression that evaluates to an integer between 8 and 65535. Must be a
+ constant. Defaults to 200.
+ """,
examples = """
Examples:
> SELECT LENGTH(kll_sketch_to_string_float(_FUNC_(col))) > 0 FROM VALUES (CAST(1.0 AS FLOAT)), (CAST(2.0 AS FLOAT)), (CAST(3.0 AS FLOAT)), (CAST(4.0 AS FLOAT)), (CAST(5.0 AS FLOAT)) tab(col);
@@ -347,9 +356,10 @@ case class KllSketchAggFloat(
arguments = """
Arguments:
* expr - The expression to aggregate into the KLL sketch.
- An expression that evaluates to a float or double.
- * k - The parameter controlling the size and accuracy of the sketch.
- An expression that evaluates to an integer. Must be a constant.
+ An expression that evaluates to a float or double.
+ * k - Optional. The parameter controlling the size and accuracy of the sketch.
+ An expression that evaluates to an integer between 8 and 65535. Must be a
+ constant. Defaults to 200.
""",
examples = """
Examples:
@@ -493,6 +503,14 @@ case class KllSketchAggDouble(
The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535).
If k is not specified, the merged sketch adopts the k value from the first input sketch.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to merge into the KLL sketch.
+ An expression that evaluates to a binary KLL sketch representation.
+ * k - Optional. The parameter controlling the size and accuracy of the merged
+ sketch. An expression that evaluates to an integer between 8 and 65535. Must be
+ a constant. Defaults to the k value of the first input sketch.
+ """,
examples = """
Examples:
> SELECT kll_sketch_get_n_bigint(_FUNC_(sketch)) FROM (SELECT kll_sketch_agg_bigint(col) as sketch FROM VALUES (1), (2), (3) tab(col) UNION ALL SELECT kll_sketch_agg_bigint(col) as sketch FROM VALUES (4), (5), (6) tab(col)) t;
@@ -566,6 +584,14 @@ case class KllMergeAggBigint(
The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535).
If k is not specified, the merged sketch adopts the k value from the first input sketch.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to merge into the KLL sketch.
+ An expression that evaluates to a binary KLL sketch representation.
+ * k - Optional. The parameter controlling the size and accuracy of the merged
+ sketch. An expression that evaluates to an integer between 8 and 65535. Must be
+ a constant. Defaults to the k value of the first input sketch.
+ """,
examples = """
Examples:
> SELECT kll_sketch_get_n_float(_FUNC_(sketch)) FROM (SELECT kll_sketch_agg_float(col) as sketch FROM VALUES (CAST(1.0 AS FLOAT)), (CAST(2.0 AS FLOAT)), (CAST(3.0 AS FLOAT)) tab(col) UNION ALL SELECT kll_sketch_agg_float(col) as sketch FROM VALUES (CAST(4.0 AS FLOAT)), (CAST(5.0 AS FLOAT)), (CAST(6.0 AS FLOAT)) tab(col)) t;
@@ -639,6 +665,14 @@ case class KllMergeAggFloat(
The optional k parameter controls the size and accuracy of the merged sketch (range 8-65535).
If k is not specified, the merged sketch adopts the k value from the first input sketch.
""",
+ arguments = """
+ Arguments:
+ * expr - The expression to merge into the KLL sketch.
+ An expression that evaluates to a binary KLL sketch representation.
+ * k - Optional. The parameter controlling the size and accuracy of the merged
+ sketch. An expression that evaluates to an integer between 8 and 65535. Must be
+ a constant. Defaults to the k value of the first input sketch.
+ """,
examples = """
Examples:
> SELECT kll_sketch_get_n_double(_FUNC_(sketch)) FROM (SELECT kll_sketch_agg_double(col) as sketch FROM VALUES (CAST(1.0 AS DOUBLE)), (CAST(2.0 AS DOUBLE)), (CAST(3.0 AS DOUBLE)) tab(col) UNION ALL SELECT kll_sketch_agg_double(col) as sketch FROM VALUES (CAST(4.0 AS DOUBLE)), (CAST(5.0 AS DOUBLE)), (CAST(6.0 AS DOUBLE)) tab(col)) t;
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
index cff1c6b9750f..4427dc25097e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala
@@ -271,6 +271,15 @@ abstract class PercentileBase
positive integral
""",
+ arguments = """
+ Arguments:
+ * col - The column to compute the percentile of.
+ An expression that evaluates to a numeric, ANSI interval, or time.
+ * percentage - The percentile(s) to compute, each between 0.0 and 1.0. Either a
+ single numeric value or an array of numeric values. Must be foldable.
+ * frequency - Optional. The number of times each value should be counted.
+ An expression that evaluates to a positive integral value. Defaults to 1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col, 0.3) FROM VALUES (0), (10) AS tab(col);
@@ -493,6 +502,12 @@ case class PercentileDisc(
usage = "_FUNC_(percentage) WITHIN GROUP (ORDER BY col) - Return a percentile value based on " +
"a continuous distribution of numeric, ANSI interval or TIME column `col` at the given " +
"`percentage` (specified in ORDER BY clause).",
+ arguments = """
+ Arguments:
+ * percentage - The percentile to compute, between 0.0 and 1.0. Must be foldable.
+ * col - The column to compute the percentile of, specified in the ORDER BY clause.
+ An expression that evaluates to a numeric, ANSI interval, or time.
+ """,
examples = """
Examples:
> SELECT _FUNC_(0.25) WITHIN GROUP (ORDER BY col) FROM VALUES (0), (10) AS tab(col);
@@ -519,6 +534,12 @@ object PercentileContBuilder extends ExpressionBuilder {
usage = "_FUNC_(percentage) WITHIN GROUP (ORDER BY col) - Return a percentile value based on " +
"a discrete distribution of numeric, ANSI interval or TIME column `col` at the given " +
"`percentage` (specified in ORDER BY clause).",
+ arguments = """
+ Arguments:
+ * percentage - The percentile to compute, between 0.0 and 1.0. Must be foldable.
+ * col - The column to compute the percentile of, specified in the ORDER BY clause.
+ An expression that evaluates to a numeric, ANSI interval, or time.
+ """,
examples = """
Examples:
> SELECT _FUNC_(0.25) WITHIN GROUP (ORDER BY col) FROM VALUES (0), (10) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
index d8cd93129ec8..a5326ae76c9f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala
@@ -323,6 +323,13 @@ case class ThetaSketchAgg(
_FUNC_(expr, lgNomEntries) - Returns the ThetaSketch's Compact binary representation.
`lgNomEntries` (optional) the log-base-2 of Nominal Entries, with Nominal Entries deciding
the number buckets or slots for the ThetaSketch.""",
+ arguments = """
+ Arguments:
+ * expr - A binary expression of Compact ThetaSketch representations to union.
+ * lgNomEntries - An optional integer expression, the log-base-2 of nominal entries which
+ sets the number of buckets for the resulting ThetaSketch. When omitted, it defaults to
+ the default log nominal entries.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(sketch)) FROM (SELECT theta_sketch_agg(col) as sketch FROM VALUES (1) tab(col) UNION ALL SELECT theta_sketch_agg(col, 20) as sketch FROM VALUES (1) tab(col));
@@ -507,6 +514,10 @@ case class ThetaUnionAgg(
usage = """
_FUNC_(expr) - Returns the ThetaSketch's Compact binary representation
by intersecting all the Theta sketches in the input column.""",
+ arguments = """
+ Arguments:
+ * expr - A binary expression of Compact ThetaSketch representations to intersect.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(sketch)) FROM (SELECT theta_sketch_agg(col) as sketch FROM VALUES (1) tab(col) UNION ALL SELECT theta_sketch_agg(col, 20) as sketch FROM VALUES (1) tab(col));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
index 03bb84c2623e..d0d7507e3250 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
@@ -502,6 +502,13 @@ object Add {
@ExpressionDescription(
usage = "expr1 _FUNC_ expr2 - Returns `expr1`-`expr2`.",
+ arguments = """
+ Arguments:
+ * expr1 - The minuend.
+ An expression that evaluates to a numeric, interval, date, timestamp, or time.
+ * expr2 - The subtrahend.
+ An expression that evaluates to a numeric, interval, date, timestamp, or time.
+ """,
examples = """
Examples:
> SELECT 2 _FUNC_ 1;
@@ -1313,6 +1320,12 @@ object Pmod {
*/
@ExpressionDescription(
usage = "_FUNC_(expr, ...) - Returns the least value of all parameters, skipping null values.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type. At least two expressions must be given,
+ and all arguments must share a common type. Null values are skipped; the result is
+ null only when all arguments are null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(10, 9, 2, 4, 3);
@@ -1402,6 +1415,12 @@ case class Least(children: Seq[Expression]) extends ComplexTypeMergingExpression
*/
@ExpressionDescription(
usage = "_FUNC_(expr, ...) - Returns the greatest value of all parameters, skipping null values.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any orderable type. At least two expressions must be given,
+ and all arguments must share a common type. Null values are skipped; the result is
+ null only when all arguments are null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(10, 9, 2, 4, 3);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
index aea04913e468..cb553aeb96ff 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala
@@ -40,6 +40,13 @@ import org.apache.spark.util.Utils
usage = """
_FUNC_(child, jsonFormatSchema, options) - Converts a binary Avro value into a Catalyst value.
""",
+ arguments = """
+ Arguments:
+ * child - A binary value containing Avro-encoded data.
+ * jsonFormatSchema - A constant string with the Avro schema in JSON format used to
+ interpret the data.
+ * options - A constant map of string keys and values holding conversion options.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}', map()) IS NULL AS result FROM (SELECT NAMED_STRUCT('u', NAMED_STRUCT('member0', member0, 'member1', member1)) AS s FROM VALUES (1, NULL), (NULL, 'a') tab(member0, member1));
@@ -141,6 +148,12 @@ case class FromAvro(child: Expression, jsonFormatSchema: Expression, options: Ex
_FUNC_(child[, jsonFormatSchema]) - Converts a Catalyst binary input value into its
corresponding Avro format result.
""",
+ arguments = """
+ Arguments:
+ * child - A value to encode into Avro binary format.
+ * jsonFormatSchema - An optional constant string with the Avro schema in JSON format
+ used to encode the value. If omitted, the schema is inferred from the input type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, '{"type": "record", "name": "struct", "fields": [{ "name": "u", "type": ["int","string"] }]}') IS NULL FROM (SELECT NULL AS s);
@@ -215,6 +228,11 @@ case class ToAvro(child: Expression, jsonFormatSchema: Expression)
usage = """
_FUNC_(jsonFormatSchema, options) - Returns schema in the DDL format of the avro schema in JSON string format.
""",
+ arguments = """
+ Arguments:
+ * jsonFormatSchema - A constant string with the Avro schema in JSON format.
+ * options - A constant map of string keys and values holding conversion options.
+ """,
examples = """
Examples:
> SELECT _FUNC_('{"type": "record", "name": "struct", "fields": [{"name": "u", "type": ["int", "string"]}]}', map());
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
index 1f943ed7edb0..87c04d229e91 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala
@@ -106,6 +106,11 @@ case class BitmapBitPosition(child: Expression)
@ExpressionDescription(
usage = "_FUNC_(child) - Returns the number of set bits in the child bitmap.",
+ arguments = """
+ Arguments:
+ * child - The bitmap whose set bits are counted. An expression that evaluates to a binary
+ bitmap, typically produced by bitmap_construct_agg().
+ """,
examples = """
Examples:
> SELECT _FUNC_(X '1010');
@@ -250,6 +255,11 @@ case class BitmapConstructAgg(child: Expression,
_FUNC_(child) - Returns a bitmap that is the bitwise OR of all of the bitmaps from the child
expression. The input should be bitmaps created from bitmap_construct_agg().
""",
+ arguments = """
+ Arguments:
+ * child - The expression whose bitmap values are combined with a bitwise OR.
+ An expression that evaluates to a binary bitmap created from bitmap_construct_agg().
+ """,
// scalastyle:off line.size.limit
examples = """
Examples:
@@ -343,6 +353,11 @@ case class BitmapOrAgg(child: Expression,
_FUNC_(child) - Returns a bitmap that is the bitwise AND of all of the bitmaps from the child
expression. The input should be bitmaps created from bitmap_construct_agg().
""",
+ arguments = """
+ Arguments:
+ * child - The expression whose bitmap values are combined with a bitwise AND.
+ An expression that evaluates to a binary bitmap created from bitmap_construct_agg().
+ """,
// scalastyle:off line.size.limit
examples = """
Examples:
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index bae2e0f6b383..4a842f2c27f9 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -191,6 +191,10 @@ case class ArraySize(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map) - Returns an unordered array containing the keys of the map.",
+ arguments = """
+ Arguments:
+ * map - A map expression whose keys are returned as an array.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'));
@@ -226,6 +230,11 @@ case class MapKeys(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map, key) - Returns true if the map contains the key.",
+ arguments = """
+ Arguments:
+ * map - A map expression to search.
+ * key - A key to look for. Its type must match, or be coercible to, the map's key type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'), 1);
@@ -488,6 +497,10 @@ object ArraysZip {
*/
@ExpressionDescription(
usage = "_FUNC_(map) - Returns an unordered array containing the values of the map.",
+ arguments = """
+ Arguments:
+ * map - A map expression whose values are returned as an array.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'));
@@ -522,6 +535,10 @@ case class MapValues(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map) - Returns an unordered array of all entries in the given map.",
+ arguments = """
+ Arguments:
+ * map - A map expression whose entries are returned as an array of key-value structs.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'));
@@ -697,6 +714,11 @@ case class MapEntries(child: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(map, ...) - Returns the union of all the given maps",
+ arguments = """
+ Arguments:
+ * map - A map expression. There can be one or more of them, and all must share
+ compatible key and value types.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'), map(3, 'c'));
@@ -825,6 +847,10 @@ case class MapConcat(children: Seq[Expression])
*/
@ExpressionDescription(
usage = "_FUNC_(arrayOfEntries) - Returns a map created from the given array of entries.",
+ arguments = """
+ Arguments:
+ * arrayOfEntries - An array of two-field key-value structs from which the map is built.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(struct(1, 'a'), struct(2, 'b')));
@@ -1621,6 +1647,12 @@ case class ArrayContains(left: Expression, right: Expression)
@ExpressionDescription(
usage = "_FUNC_(array, value) - Return index (0-based) of the search value, " +
"if it is contained in the array; otherwise, (-<insertion point> - 1).",
+ arguments = """
+ Arguments:
+ * array - A sorted array expression to search in.
+ * value - The value to search for. Its type must match, or be coercible to, the
+ array's element type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1, 2, 3), 2);
@@ -3022,6 +3054,11 @@ case class TryElementAt(left: Expression, right: Expression, replacement: Expres
*/
@ExpressionDescription(
usage = "_FUNC_(col1, col2, ..., colN) - Returns the concatenation of col1, col2, ..., colN.",
+ arguments = """
+ Arguments:
+ * colN - An expression to concatenate. There can be one or more of them, and all must be
+ of the same type: strings, binaries, or arrays.
+ """,
examples = """
Examples:
> SELECT _FUNC_('Spark', 'SQL');
@@ -3243,6 +3280,11 @@ case class Concat(children: Seq[Expression]) extends ComplexTypeMergingExpressio
*/
@ExpressionDescription(
usage = "_FUNC_(arrayOfArrays) - Transforms an array of arrays into a single array.",
+ arguments = """
+ Arguments:
+ * arrayOfArrays - An array whose elements are themselves arrays; they are concatenated
+ in order into one array.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(array(1, 2), array(3, 4)));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
index 122272ba4333..8629f900ac43 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
@@ -55,6 +55,11 @@ trait NoThrow
*/
@ExpressionDescription(
usage = "_FUNC_(expr, ...) - Returns an array with the given elements.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type to include as an array element. Zero or more
+ expressions can be given, and they must share a common type.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1, 2, 3);
@@ -179,6 +184,14 @@ private [sql] object GenArrayData {
*/
@ExpressionDescription(
usage = "_FUNC_(key0, value0, key1, value1, ...) - Creates a map with the given key/value pairs.",
+ arguments = """
+ Arguments:
+ * keyN - An expression for the N-th map key. Keys must not be NULL and all keys must
+ share a common type.
+ * valueN - An expression for the value paired with keyN. All values must share a common
+ type. Keys and values are supplied as alternating arguments, so the total number of
+ arguments must be even.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1.0, '2', 3.0, '4');
@@ -423,7 +436,12 @@ object CreateStruct {
null,
"struct",
"_FUNC_(col1, col2, col3, ...) - Creates a struct with the given field values.",
- "",
+ """
+ | Arguments:
+ | * colN - The field values of the struct. There can be zero or more of them,
+ | each an expression of any type. Field names are assigned as `colN` by
+ | default unless the value is a named expression.
+ | """.stripMargin,
"""
| Examples:
| > SELECT _FUNC_(1, 2, 3);
@@ -446,6 +464,13 @@ object CreateStruct {
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(name1, val1, name2, val2, ...) - Creates a struct with the given field names and values.",
+ arguments = """
+ Arguments:
+ * nameN - A STRING literal giving the name of the N-th struct field. It must not be NULL.
+ * valN - An expression of any type providing the value for the field named nameN. Names
+ and values are supplied as alternating arguments, so the total number of arguments
+ must be even.
+ """,
examples = """
Examples:
> SELECT _FUNC_("a", 1, "b", 2, "c", 3);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
index 621f02ca18b8..24dd3623cd13 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
@@ -33,6 +33,12 @@ import org.apache.spark.util.ArrayImplicits._
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, expr3) - If `expr1` evaluates to true, then returns `expr2`; otherwise returns `expr3`.",
+ arguments = """
+ Arguments:
+ * expr1 - A boolean expression evaluated as the condition.
+ * expr2 - The expression returned when `expr1` evaluates to true.
+ * expr3 - The expression returned when `expr1` evaluates to false or null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1 < 2, 'a', 'b');
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
index 5110aab9c6f0..ae10f0245141 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala
@@ -40,6 +40,14 @@ import org.apache.spark.unsafe.types.UTF8String
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(csvStr, schema[, options]) - Returns a struct value with the given `csvStr` and `schema`.",
+ arguments = """
+ Arguments:
+ * csvStr - A string expression of a single CSV record.
+ * schema - A string literal or invocation of `schema_of_csv` describing the schema.
+ * options - An optional map literal of string key-value pairs specifying CSV parsing
+ options controlling how `csvStr` is parsed. Accepts the same options as the CSV data
+ source.
+ """,
examples = """
Examples:
> SELECT _FUNC_('1, 0.8', 'a INT, b DOUBLE');
@@ -133,6 +141,12 @@ case class CsvToStructs(
*/
@ExpressionDescription(
usage = "_FUNC_(csv[, options]) - Returns schema in the DDL format of CSV string.",
+ arguments = """
+ Arguments:
+ * csv - A foldable string expression of a single CSV record.
+ * options - An optional map literal of string key-value pairs specifying CSV parsing
+ options that control schema inference. Accepts the same options as the CSV data source.
+ """,
examples = """
Examples:
> SELECT _FUNC_('1,abc');
@@ -205,6 +219,13 @@ case class SchemaOfCsv(
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr[, options]) - Returns a CSV string with a given struct value",
+ arguments = """
+ Arguments:
+ * expr - A struct expression to convert into a CSV string.
+ * options - An optional map literal of string key-value pairs specifying CSV generation
+ options controlling how the struct is rendered. Accepts the same options as the CSV data
+ source.
+ """,
examples = """
Examples:
> SELECT _FUNC_(named_struct('a', 1, 'b', 2));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
index a9baf473c822..743dd27c9240 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datasketchesExpressions.scala
@@ -29,6 +29,11 @@ import org.apache.spark.sql.types.{AbstractDataType, BinaryType, BooleanType, Da
usage = """
_FUNC_(expr) - Returns the estimated number of unique values given the binary representation
of a Datasketches HllSketch. """,
+ arguments = """
+ Arguments:
+ * expr - A binary expression holding the serialized representation of a Datasketches
+ HllSketch.
+ """,
examples = """
Examples:
> SELECT _FUNC_(hll_sketch_agg(col)) FROM VALUES (1), (1), (2), (2), (3) tab(col);
@@ -70,6 +75,15 @@ case class HllSketchEstimate(child: Expression)
Datasketches HllSketch objects, using a Datasketches Union object. Set
allowDifferentLgConfigK to true to allow unions of sketches with different
lgConfigK values (defaults to false). """,
+ arguments = """
+ Arguments:
+ * first - A binary expression holding the serialized representation of a Datasketches
+ HllSketch.
+ * second - A binary expression holding the serialized representation of a Datasketches
+ HllSketch.
+ * allowDifferentLgConfigK - A boolean. Set to true to allow unions of sketches with
+ different lgConfigK values. Defaults to false.
+ """,
examples = """
Examples:
> SELECT hll_sketch_estimate(_FUNC_(hll_sketch_agg(col1), hll_sketch_agg(col2))) FROM VALUES (1, 4), (1, 4), (2, 5), (2, 5), (3, 6) tab(col1, col2);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index df9b4571440f..69fa3e8fbc18 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -1126,6 +1126,10 @@ case class UnixMicros(child: Expression) extends TimestampToLongBase {
// scalastyle:off line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(timestamp) - Returns the number of nanoseconds since 1970-01-01 00:00:00 UTC.",
+ arguments = """
+ Arguments:
+ * timestamp - A nanosecond-precision timestamp value (TIMESTAMP_LTZ or TIMESTAMP_NTZ).
+ """,
examples = """
Examples:
> SET spark.sql.timestampNanosTypes.enabled=true;
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
index de8d40cd4299..3f2e838c1903 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala
@@ -143,6 +143,13 @@ case class UserDefinedGenerator(
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(n, expr1, ..., exprk) - Separates `expr1`, ..., `exprk` into `n` rows. Uses column names col0, col1, etc. by default unless specified otherwise.",
+ arguments = """
+ Arguments:
+ * n - The number of rows to separate the expressions into. Must be a
+ positive integer literal.
+ * exprN - The expressions to separate into rows. Their values are laid out
+ row-major across the `n` output rows.
+ """,
examples = """
Examples:
> SELECT _FUNC_(2, 1, 2, 3);
@@ -437,6 +444,10 @@ trait ExplodeGeneratorBuilderBase extends GeneratorBuilder {
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple rows, or the elements of map `expr` into multiple rows and columns. Unless specified otherwise, uses the default column name `col` for elements of the array or `key` and `value` for the elements of the map.",
+ arguments = """
+ Arguments:
+ * expr - An array or map expression whose elements are separated into rows.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(10, 20));
@@ -522,6 +533,11 @@ trait PosExplodeGeneratorBuilderBase extends GeneratorBuilder {
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(expr) - Separates the elements of array `expr` into multiple rows with positions, or the elements of map `expr` into multiple rows and columns with positions. Unless specified otherwise, uses the column name `pos` for position, `col` for elements of the array or `key` and `value` for elements of the map.",
+ arguments = """
+ Arguments:
+ * expr - An array or map expression whose elements are separated into rows,
+ each paired with its position.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(10,20));
@@ -643,6 +659,11 @@ trait InlineGeneratorBuilderBase extends GeneratorBuilder {
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(expr) - Explodes an array of structs into a table. Uses column names col1, col2, etc. by default unless specified otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An array-of-structs expression whose struct fields become the
+ columns of each generated row.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(struct(1, 'a'), struct(2, 'b')));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
index 0a12735b38da..adb26487b1bf 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/grouping.scala
@@ -196,6 +196,11 @@ object GroupingSets {
_FUNC_(col) - indicates whether a specified column in a GROUP BY is aggregated or
not, returns 1 for aggregated or 0 for not aggregated in the result set.",
""",
+ arguments = """
+ Arguments:
+ * col - A grouping column referenced in the GROUP BY clause. It must exactly match one
+ of the grouping expressions of the query.
+ """,
examples = """
Examples:
> SELECT name, _FUNC_(name), sum(age) FROM VALUES (2, 'Alice'), (5, 'Bob') people(age, name) GROUP BY cube(name);
@@ -229,6 +234,12 @@ case class Grouping(child: Expression) extends Expression with Unevaluable
_FUNC_([col1[, col2 ..]]) - returns the level of grouping, equals to
`(grouping(c1) << (n-1)) + (grouping(c2) << (n-2)) + ... + grouping(cn)`
""",
+ arguments = """
+ Arguments:
+ * colN - An optional grouping column referenced in the GROUP BY clause. Zero or more
+ columns can be given; when provided, they must match the grouping columns exactly,
+ and when omitted all grouping columns are used.
+ """,
examples = """
Examples:
> SELECT name, _FUNC_(), sum(age), avg(height) FROM VALUES (2, 'Alice', 165), (5, 'Bob', 180) people(age, name, height) GROUP BY cube(name, height);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
index 085586041b07..82c273804d23 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala
@@ -762,6 +762,11 @@ abstract class InterpretedHashFunction {
*/
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, ...) - Returns a hash value of the arguments.",
+ arguments = """
+ Arguments:
+ * exprN - The values to hash. There can be one or more of them, each an
+ expression of any data type.
+ """,
examples = """
Examples:
> SELECT _FUNC_('Spark', array(123), 2);
@@ -833,6 +838,11 @@ case class CollationAwareMurmur3Hash(children: Seq[Expression], seed: Int)
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, ...) - Returns a 64-bit hash value of the arguments. " +
"Hash seed is 42.",
+ arguments = """
+ Arguments:
+ * exprN - The values to hash. There can be one or more of them, each an
+ expression of any data type.
+ """,
examples = """
Examples:
> SELECT _FUNC_('Spark', array(123), 2);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
index 524e27321f4f..cc34421b6230 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala
@@ -658,6 +658,12 @@ object ArraySort {
*/
@ExpressionDescription(
usage = "_FUNC_(expr, func) - Filters entries in a map using the function.",
+ arguments = """
+ Arguments:
+ * expr - A map expression.
+ * func - A lambda function `(k, v) -> boolean` that returns whether the entry with
+ key `k` and value `v` should be kept.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 0, 2, 2, 3, -1), (k, v) -> k > v);
@@ -1352,6 +1358,12 @@ case class ArrayAggregate(
*/
@ExpressionDescription(
usage = "_FUNC_(expr, func) - Transforms elements in a map using the function.",
+ arguments = """
+ Arguments:
+ * expr - A map expression.
+ * func - A lambda function `(k, v) -> newKey` producing the transformed key from the
+ entry with key `k` and value `v`. The values are kept unchanged.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> k + 1);
@@ -1412,6 +1424,12 @@ case class TransformKeys(
*/
@ExpressionDescription(
usage = "_FUNC_(expr, func) - Transforms values in the map using the function.",
+ arguments = """
+ Arguments:
+ * expr - A map expression.
+ * func - A lambda function `(k, v) -> newValue` producing the transformed value from
+ the entry with key `k` and value `v`. The keys are kept unchanged.
+ """,
examples = """
Examples:
> SELECT _FUNC_(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, v) -> v + 1);
@@ -1472,6 +1490,14 @@ case class TransformValues(
NULL will be passed as the value for the missing key. If an input map contains duplicated
keys, only the first entry of the duplicated key is passed into the lambda function.
""",
+ arguments = """
+ Arguments:
+ * map1 - The first map expression.
+ * map2 - The second map expression.
+ * function - A lambda function `(k, v1, v2) -> newValue` that produces the merged value
+ for key `k`, where `v1` and `v2` are the values from `map1` and `map2` respectively
+ (NULL when the key is missing from that map).
+ """,
examples = """
Examples:
> SELECT _FUNC_(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> concat(v1, v2));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
index d8131c3afb42..1621ec9ba9ac 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
@@ -244,6 +244,12 @@ case class MultiGetJsonObject(
// scalastyle:off line.size.limit line.contains.tab
@ExpressionDescription(
usage = "_FUNC_(jsonStr, p1, p2, ..., pn) - Returns a tuple like the function get_json_object, but it takes multiple names. All the input parameters and output column types are string.",
+ arguments = """
+ Arguments:
+ * jsonStr - A JSON string to extract fields from.
+ * pN - The field names to extract. Each name yields one output column with
+ the corresponding field value.
+ """,
examples = """
Examples:
> SELECT _FUNC_('{"a":1, "b":2}', 'a', 'b');
@@ -346,6 +352,14 @@ case class JsonTuple(children: Seq[Expression])
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(jsonStr, schema[, options]) - Returns a struct value with the given `jsonStr` and `schema`.",
+ arguments = """
+ Arguments:
+ * jsonStr - A JSON string to parse.
+ * schema - The schema to use when parsing the JSON string, given as a DDL
+ formatted string or a schema expression.
+ * options - Optional. A map of string key-value pairs that control how the
+ JSON is parsed. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_('{"a":1, "b":0.8}', 'a INT, b DOUBLE');
@@ -549,6 +563,12 @@ case class StructsToJson(
*/
@ExpressionDescription(
usage = "_FUNC_(json[, options]) - Returns schema in the DDL format of JSON string.",
+ arguments = """
+ Arguments:
+ * json - A JSON string whose schema is inferred.
+ * options - Optional. A map of string key-value pairs that control how the
+ JSON is parsed. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_('[{"col":0}]');
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
index e80277666109..fa456d8955bf 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
@@ -364,6 +364,10 @@ case class SparkVersion()
@ExpressionDescription(
usage = """_FUNC_(expr) - Return DDL-formatted type string for the data type of the input.""",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type whose data type is returned as a DDL-formatted string.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
index eb8e67eca3f1..db77d6b82b9f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala
@@ -41,6 +41,11 @@ import org.apache.spark.sql.types._
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, ...) - Returns the first non-null argument if exists. Otherwise, null.",
+ arguments = """
+ Arguments:
+ * exprN - An expression of any type. All arguments must share a common type.
+ Arguments are evaluated in order and the first non-null value is returned.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL, 1, NULL);
@@ -252,6 +257,10 @@ case class NullIfZero(input: Expression, replacement: Expression)
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns zero if `expr` is equal to null, or `expr` otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An expression. Zero is returned when it is null, otherwise `expr` is returned.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL);
@@ -273,6 +282,11 @@ case class ZeroIfNull(input: Expression, replacement: Expression)
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2) - Returns `expr2` if `expr1` is null, or `expr1` otherwise.",
+ arguments = """
+ Arguments:
+ * expr1 - An expression. Returned when it is not null.
+ * expr2 - The value returned when `expr1` is null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL, array('2'));
@@ -297,6 +311,12 @@ case class Nvl(left: Expression, right: Expression, replacement: Expression)
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr1, expr2, expr3) - Returns `expr2` if `expr1` is not null, or `expr3` otherwise.",
+ arguments = """
+ Arguments:
+ * expr1 - An expression tested for nullability.
+ * expr2 - The value returned when `expr1` is not null.
+ * expr3 - The value returned when `expr1` is null.
+ """,
examples = """
Examples:
> SELECT _FUNC_(NULL, 2, 1);
@@ -464,6 +484,10 @@ case class NaNvl(left: Expression, right: Expression)
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns true if `expr` is null, or false otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type to test for nullability.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1);
@@ -498,6 +522,10 @@ case class IsNull(child: Expression) extends UnaryExpression with Predicate {
*/
@ExpressionDescription(
usage = "_FUNC_(expr) - Returns true if `expr` is not null, or false otherwise.",
+ arguments = """
+ Arguments:
+ * expr - An expression of any type to test for nullability.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index 2e2f30330094..6899ff38f996 100755
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -280,6 +280,12 @@ case class ConcatWs(children: Seq[Expression])
and `spark.sql.ansi.enabled` is set to false. If `spark.sql.ansi.enabled` is set to true,
it throws ArrayIndexOutOfBoundsException for invalid indices.
""",
+ arguments = """
+ Arguments:
+ * n - An integer expression giving the 1-based index of the input to return.
+ * input1, input2, ... - The input expressions to select from. They can be strings
+ or binary values.
+ """,
examples = """
Examples:
> SELECT _FUNC_(1, 'scala', 'java');
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
index 8ac40a3fe2a5..1aa668a9b894 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/thetasketchesExpressions.scala
@@ -28,6 +28,11 @@ import org.apache.spark.sql.types.{AbstractDataType, BinaryType, DataType, Integ
usage = """
_FUNC_(expr) - Returns the estimated number of unique values
given the binary representation of a Datasketches ThetaSketch. """,
+ arguments = """
+ Arguments:
+ * expr - The binary representation of a Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(theta_sketch_agg(col)) FROM VALUES (1), (1), (2), (2), (3) tab(col);
@@ -66,6 +71,16 @@ case class ThetaSketchEstimate(child: Expression)
Datasketches ThetaSketch objects using a ThetaSketch Union object. Users can set
lgNomEntries to a value between 4 and 26 to find the union of sketches with different
union buffer size values (defaults to 12). """,
+ arguments = """
+ Arguments:
+ * first - The binary representation of the first Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ * second - The binary representation of the second Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ * lgNomEntries - Optional. The log-base-2 of the nominal entries used to size the
+ union buffer, between 4 and 26. An expression that evaluates to an integer.
+ Defaults to 12.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(theta_sketch_agg(col1), theta_sketch_agg(col2))) FROM VALUES (1, 4), (1, 4), (2, 5), (2, 5), (3, 6) tab(col1, col2);
@@ -125,6 +140,13 @@ case class ThetaUnion(first: Expression, second: Expression, third: Expression)
_FUNC_(first, second) - Subtracts two binary representations of
Datasketches ThetaSketch objects from two input columns using a
ThetaSketch AnotB object. """,
+ arguments = """
+ Arguments:
+ * first - The binary representation of the first Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ * second - The binary representation of the second Datasketches ThetaSketch to
+ subtract from the first. An expression that evaluates to binary.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(theta_sketch_agg(col1), theta_sketch_agg(col2))) FROM VALUES (5, 4), (1, 4), (2, 5), (2, 5), (3, 1) tab(col1, col2);
@@ -173,6 +195,13 @@ case class ThetaDifference(first: Expression, second: Expression)
_FUNC_(first, second) - Intersects two binary representations of
Datasketches ThetaSketch objects from two input columns using a
ThetaSketch Intersect object. """,
+ arguments = """
+ Arguments:
+ * first - The binary representation of the first Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ * second - The binary representation of the second Datasketches ThetaSketch.
+ An expression that evaluates to binary.
+ """,
examples = """
Examples:
> SELECT theta_sketch_estimate(_FUNC_(theta_sketch_agg(col1), theta_sketch_agg(col2))) FROM VALUES (5, 4), (1, 4), (2, 5), (2, 5), (3, 1) tab(col1, col2);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
index 7d0fa35b3f0f..644695dfabbf 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala
@@ -46,6 +46,18 @@ import org.apache.spark.util.Utils
usage = """
_FUNC_(data, messageName, descFilePath, options) - Converts a binary Protobuf value into a Catalyst value.
""",
+ arguments = """
+ Arguments:
+ * data - The binary Protobuf value to convert.
+ * messageName - A constant string naming the Protobuf message to look for in
+ the descriptor file.
+ * descFilePath - Optional. A constant string or binary value with the
+ Protobuf descriptor file, created by `protoc` with `--descriptor_set_out`
+ and `--include_imports`. If omitted, the message must be resolvable
+ otherwise.
+ * options - Optional. A constant map of string key-value pairs controlling
+ the conversion. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, 'Person', '/path/to/descriptor.desc', map()) IS NULL AS result FROM (SELECT NAMED_STRUCT('name', name, 'id', id) AS s FROM VALUES ('John Doe', 1), (NULL, 2) tab(name, id));
@@ -190,6 +202,17 @@ case class FromProtobuf(
_FUNC_(child, messageName, descFilePath, options) - Converts a Catalyst binary input value into its corresponding
Protobuf format result.
""",
+ arguments = """
+ Arguments:
+ * child - The Catalyst input value to convert to Protobuf binary.
+ * messageName - A constant string naming the Protobuf message to serialize to.
+ * descFilePath - Optional. A constant string or binary value with the
+ Protobuf descriptor file, created by `protoc` with `--descriptor_set_out`
+ and `--include_imports`. If omitted, the message must be resolvable
+ otherwise.
+ * options - Optional. A constant map of string key-value pairs controlling
+ the conversion. By default no options are set.
+ """,
examples = """
Examples:
> SELECT _FUNC_(s, 'Person', '/path/to/descriptor.desc', map('emitDefaultValues', 'true')) IS NULL FROM (SELECT NULL AS s);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
index e26354b54f6c..539af821e8e8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleDifference.scala
@@ -31,6 +31,12 @@ import org.apache.spark.sql.types.{AbstractDataType, BinaryType, DataType}
_FUNC_(tupleSketch1, tupleSketch2) - Subtracts two binary representations of Datasketches
TupleSketch objects with double summary data type using a TupleSketch AnotB object.
Returns elements in the first sketch that are not in the second sketch. """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - A binary representation of a TupleSketch with double summary.
+ * tupleSketch2 - A binary representation of a TupleSketch with double summary
+ to subtract from the first sketch.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), tuple_sketch_agg_double(col2, val2))) FROM VALUES (5, 5.0D, 4, 4.0D), (1, 1.0D, 4, 4.0D), (2, 2.0D, 5, 5.0D), (3, 3.0D, 1, 1.0D) tab(col1, val1, col2, val2);
@@ -67,6 +73,12 @@ case class TupleDifferenceDouble(left: Expression, right: Expression)
_FUNC_(tupleSketch1, tupleSketch2) - Subtracts two binary representations of Datasketches
TupleSketch objects with integer summary data type using a TupleSketch AnotB object.
Returns elements in the first sketch that are not in the second sketch. """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - A binary representation of a TupleSketch with integer summary.
+ * tupleSketch2 - A binary representation of a TupleSketch with integer summary
+ to subtract from the first sketch.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1), tuple_sketch_agg_integer(col2, val2))) FROM VALUES (5, 5, 4, 4), (1, 1, 4, 4), (2, 2, 5, 5), (3, 3, 1, 1) tab(col1, val1, col2, val2);
@@ -103,6 +115,11 @@ case class TupleDifferenceInteger(left: Expression, right: Expression)
_FUNC_(tupleSketch, thetaSketch) - Subtracts the binary representation of a
Datasketches ThetaSketch from a TupleSketch with double summary data type using a TupleSketch
AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch. """,
+ arguments = """
+ Arguments:
+ * tupleSketch - A binary representation of a TupleSketch with double summary.
+ * thetaSketch - A binary representation of a ThetaSketch to subtract from the TupleSketch.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), theta_sketch_agg(col2))) FROM VALUES (5, 5.0D, 4), (1, 1.0D, 4), (2, 2.0D, 5), (3, 3.0D, 1) tab(col1, val1, col2);
@@ -139,6 +156,11 @@ case class TupleDifferenceThetaDouble(left: Expression, right: Expression)
_FUNC_(tupleSketch, thetaSketch) - Subtracts the binary representation of a
Datasketches ThetaSketch from a TupleSketch with integer summary data type using a TupleSketch
AnotB object. Returns elements in the TupleSketch that are not in the ThetaSketch. """,
+ arguments = """
+ Arguments:
+ * tupleSketch - A binary representation of a TupleSketch with integer summary.
+ * thetaSketch - A binary representation of a ThetaSketch to subtract from the TupleSketch.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1), theta_sketch_agg(col2))) FROM VALUES (5, 5, 4), (1, 1, 4), (2, 2, 5), (3, 3, 1) tab(col1, val1, col2);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
index 2085b759af09..b6a5cb1d38ce 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleIntersection.scala
@@ -34,6 +34,15 @@ import org.apache.spark.unsafe.types.UTF8String
_FUNC_(tupleSketch1, tupleSketch2, mode) - Intersects two binary representations of Datasketches
TupleSketch objects with double summary data type using a TupleSketch Intersection object.
Users can set mode to 'sum', 'min', 'max', or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The binary representation of a Datasketches TupleSketch with a double
+ summary data type.
+ * tupleSketch2 - The binary representation of a Datasketches TupleSketch with a double
+ summary data type.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or 'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), tuple_sketch_agg_double(col2, val2))) FROM VALUES (1, 1.0D, 1, 4.0D), (2, 2.0D, 2, 5.0D), (3, 3.0D, 4, 6.0D) tab(col1, val1, col2, val2);
@@ -80,6 +89,15 @@ case class TupleIntersectionDouble(first: Expression, second: Expression, third:
_FUNC_(tupleSketch1, tupleSketch2, mode) - Intersects two binary representations of Datasketches
TupleSketch objects with integer summary data type using a TupleSketch Intersection object.
Users can set mode to 'sum', 'min', 'max', or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The binary representation of a Datasketches TupleSketch with an integer
+ summary data type.
+ * tupleSketch2 - The binary representation of a Datasketches TupleSketch with an integer
+ summary data type.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or 'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1), tuple_sketch_agg_integer(col2, val2))) FROM VALUES (1, 1, 1, 4), (2, 2, 2, 5), (3, 3, 4, 6) tab(col1, val1, col2, val2);
@@ -131,6 +149,14 @@ case class TupleIntersectionInteger(first: Expression, second: Expression, third
assigned a default double summary value based on the mode: 0.0 for 'sum' mode, +Infinity for
'min' mode, -Infinity for 'max' mode, or 1.0 for 'alwaysone' mode. Users can set mode to 'sum',
'min', 'max', or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The binary representation of a Datasketches TupleSketch with a double
+ summary data type.
+ * thetaSketch - The binary representation of a Datasketches ThetaSketch.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or 'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), theta_sketch_agg(col2))) FROM VALUES (1, 1.0D, 1), (2, 2.0D, 2), (3, 3.0D, 4) tab(col1, val1, col2);
@@ -182,6 +208,14 @@ case class TupleIntersectionThetaDouble(first: Expression, second: Expression, t
assigned a default integer summary value based on the mode: 0 for 'sum' mode, Integer.MAX_VALUE
for 'min' mode, Integer.MIN_VALUE for 'max' mode, or 1 for 'alwaysone' mode. Users can set mode
to 'sum', 'min', 'max', or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The binary representation of a Datasketches TupleSketch with an integer
+ summary data type.
+ * thetaSketch - The binary representation of a Datasketches ThetaSketch.
+ * mode - The summary combination mode: 'sum', 'min', 'max', or 'alwaysone'
+ (optional, defaults to 'sum').
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1), theta_sketch_agg(col2))) FROM VALUES (1, 1, 1), (2, 2, 2), (3, 3, 4) tab(col1, val1, col2);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
index fcd0a048479b..5ac9504bfa92 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchEstimate.scala
@@ -27,6 +27,11 @@ import org.apache.spark.sql.types.{AbstractDataType, BinaryType, DataType, Doubl
_FUNC_(child) - Returns the estimated number of unique values
given the binary representation of a Datasketches TupleSketch. The sketch's
summary type must be a double. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with a double summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_double(key, summary)) FROM VALUES (1, 1.0D), (1, 2.0D), (2, 3.0D) tab(key, summary);
@@ -64,6 +69,11 @@ case class TupleSketchEstimateDouble(child: Expression)
_FUNC_(child) - Returns the estimated number of unique values
given the binary representation of a Datasketches TupleSketch. The sketch's
summary type must be an integer. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with an integer summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_integer(key, summary)) FROM VALUES (1, 1), (1, 2), (2, 3) tab(key, summary);
@@ -101,6 +111,11 @@ case class TupleSketchEstimateInteger(child: Expression)
_FUNC_(child) - Returns the theta value (sampling rate) from a Datasketches TupleSketch.
The theta value represents the effective sampling rate of the sketch, between 0.0 and 1.0.
The sketch's summary type must be a double. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with a double summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_double(key, summary)) FROM VALUES (1, 1.0D), (2, 2.0D), (3, 3.0D) tab(key, summary);
@@ -138,6 +153,11 @@ case class TupleSketchThetaDouble(child: Expression)
_FUNC_(child) - Returns the theta value (sampling rate) from a Datasketches TupleSketch.
The theta value represents the effective sampling rate of the sketch, between 0.0 and 1.0.
The sketch's summary type must be an integer. """,
+ arguments = """
+ Arguments:
+ * child - A binary value holding a serialized Datasketches TupleSketch
+ with an integer summary.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_integer(key, summary)) FROM VALUES (1, 1), (2, 2), (3, 3) tab(key, summary);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
index a37a03e41e53..625ab10c3119 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleSketchSummary.scala
@@ -34,6 +34,13 @@ import org.apache.spark.unsafe.types.UTF8String
_FUNC_(child, mode) - Aggregates the summary values from a double summary type
Datasketches TupleSketch. The mode can be 'sum', 'min', 'max', or 'alwaysone'
(defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * child - A binary expression holding a serialized double summary type Datasketches
+ TupleSketch to aggregate.
+ * mode - An optional string aggregation mode: 'sum', 'min', 'max', or 'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_double(key, summary)) FROM VALUES (1, 1.0D), (1, 2.0D), (2, 3.0D) tab(key, summary);
@@ -98,6 +105,13 @@ case class TupleSketchSummaryDouble(left: Expression, right: Expression)
_FUNC_(child, mode) - Aggregates the summary values from a integer summary type
Datasketches TupleSketch. The mode can be 'sum', 'min', 'max', or 'alwaysone'
(defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * child - A binary expression holding a serialized integer summary type Datasketches
+ TupleSketch to aggregate.
+ * mode - An optional string aggregation mode: 'sum', 'min', 'max', or 'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT _FUNC_(tuple_sketch_agg_integer(key, summary)) FROM VALUES (1, 1), (1, 2), (2, 3) tab(key, summary);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
index be04fb20dd30..d174e392bb93 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala
@@ -377,6 +377,15 @@ abstract class TupleUnionBase[S <: Summary]
TupleSketch objects with double summary data type using a TupleSketch Union object. Users can
set lgNomEntries to a value between 4 and 26 (defaults to 12) and mode to 'sum', 'min', 'max',
or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The first TupleSketch (double summary) as a binary value.
+ * tupleSketch2 - The second TupleSketch (double summary) as a binary value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base 2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or 'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), tuple_sketch_agg_double(col2, val2))) FROM VALUES (1, 1.0D, 4, 4.0D), (2, 2.0D, 5, 5.0D), (3, 3.0D, 6, 6.0D) tab(col1, val1, col2, val2);
@@ -407,6 +416,15 @@ object TupleUnionDoubleExpressionBuilder extends ExpressionBuilder {
TupleSketch objects with integer summary data type using a TupleSketch Union object. Users can
set lgNomEntries to a value between 4 and 26 (defaults to 12) and mode to 'sum', 'min', 'max',
or 'alwaysone' (defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch1 - The first TupleSketch (integer summary) as a binary value.
+ * tupleSketch2 - The second TupleSketch (integer summary) as a binary value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base 2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or 'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1), tuple_sketch_agg_integer(col2, val2))) FROM VALUES (1, 1, 4, 4), (2, 2, 5, 5), (3, 3, 6, 6) tab(col1, val1, col2, val2);
@@ -440,6 +458,15 @@ object TupleUnionIntegerExpressionBuilder extends ExpressionBuilder {
-Infinity for 'max' mode, or 1.0 for 'alwaysone' mode. Users can set lgNomEntries to a value
between 4 and 26 (defaults to 12) and mode to 'sum', 'min', 'max', or 'alwaysone' (defaults to
'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The TupleSketch (double summary) as a binary value.
+ * thetaSketch - The ThetaSketch as a binary value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base 2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or 'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_double(_FUNC_(tuple_sketch_agg_double(col1, val1), theta_sketch_agg(col2))) FROM VALUES (1, 1.0D, 4), (2, 2.0D, 5), (3, 3.0D, 6) tab(col1, val1, col2);
@@ -473,6 +500,15 @@ object TupleUnionThetaDoubleExpressionBuilder extends ExpressionBuilder {
mode, Integer.MIN_VALUE for 'max' mode, or 1 for 'alwaysone' mode. Users can set lgNomEntries to
a value between 4 and 26 (defaults to 12) and mode to 'sum', 'min', 'max', or 'alwaysone'
(defaults to 'sum'). """,
+ arguments = """
+ Arguments:
+ * tupleSketch - The TupleSketch (integer summary) as a binary value.
+ * thetaSketch - The ThetaSketch as a binary value.
+ * lgNomEntries - Optional integer between 4 and 26 setting the log base 2 of the
+ number of nominal entries. Defaults to 12.
+ * mode - Optional summary combining mode, one of 'sum', 'min', 'max', or 'alwaysone'.
+ Defaults to 'sum'.
+ """,
examples = """
Examples:
> SELECT tuple_sketch_estimate_integer(_FUNC_(tuple_sketch_agg_integer(col1, val1), theta_sketch_agg(col2))) FROM VALUES (1, 1, 4), (2, 2, 5), (3, 3, 6) tab(col1, val1, col2);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
index 3448bfee62e1..61a67bf92648 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
@@ -81,6 +81,10 @@ case class ParseJson(child: Expression, failOnError: Boolean = true)
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Check if a variant value is a variant null. Returns true if and only if the input is a variant null and false otherwise (including in the case of SQL NULL).",
+ arguments = """
+ Arguments:
+ * expr - A variant value to check.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('null'));
@@ -122,6 +126,11 @@ case class IsVariantNull(child: Expression) extends UnaryExpression
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr) - Convert a nested input (array/map/struct) into a variant where maps and structs are converted to variant objects which are unordered unlike SQL structs. Input maps can only have string keys.",
+ arguments = """
+ Arguments:
+ * expr - A nested value of array, map, or struct type. Maps must have string keys. Nested
+ values of any type are allowed.
+ """,
examples = """
Examples:
> SELECT _FUNC_(named_struct('a', 1, 'b', 2));
@@ -638,6 +647,13 @@ abstract class VariantGetExpressionBuilderBase(failOnError: Boolean) extends Exp
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(v, path[, type]) - Extracts a sub-variant from `v` according to `path`, and then cast the sub-variant to `type`. When `type` is omitted, it is default to `variant`. Returns null if the path does not exist. Throws an exception if the cast fails.",
+ arguments = """
+ Arguments:
+ * v - A variant value to extract from.
+ * path - A string literal in JSONPath format that identifies the sub-variant to extract.
+ * type - An optional string literal naming the SQL type to cast the extracted sub-variant to.
+ When omitted, it defaults to `variant`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('{"a": 1}'), '$.a', 'int');
@@ -660,6 +676,13 @@ object VariantGetExpressionBuilder extends VariantGetExpressionBuilderBase(true)
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(v, path[, type]) - Extracts a sub-variant from `v` according to `path`, and then cast the sub-variant to `type`. When `type` is omitted, it is default to `variant`. Returns null if the path does not exist or the cast fails.",
+ arguments = """
+ Arguments:
+ * v - A variant value to extract from.
+ * path - A string literal in JSONPath format that identifies the sub-variant to extract.
+ * type - An optional string literal naming the SQL type to cast the extracted sub-variant to.
+ When omitted, it defaults to `variant`.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('{"a": 1}'), '$.a', 'int');
@@ -1584,6 +1607,10 @@ object VariantExplode {
@ExpressionDescription(
usage = "_FUNC_(v) - Returns schema in the SQL format of a variant.",
+ arguments = """
+ Arguments:
+ * v - A variant value whose schema is returned.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('null'));
@@ -1699,6 +1726,10 @@ object SchemaOfVariant {
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(v) - Returns the merged schema in the SQL format of a variant column.",
+ arguments = """
+ Arguments:
+ * v - A variant column whose per-row schemas are merged into a single schema.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json(j)) FROM VALUES ('1'), ('2'), ('3') AS tab(j);
@@ -1763,6 +1794,10 @@ case class SchemaOfVariantAgg(
@ExpressionDescription(
usage = "_FUNC_(v) - Returns true if the variant is valid, false if it is malformed, " +
"NULL if `v` is NULL.",
+ arguments = """
+ Arguments:
+ * v - A variant value to validate.
+ """,
examples = """
Examples:
> SELECT _FUNC_(parse_json('null'));
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
index e65fae3a2bc2..d730c8f0be7b 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
@@ -43,6 +43,12 @@ import org.apache.spark.unsafe.Platform
_FUNC_(array1, array2) - Returns the cosine similarity between two float vectors.
The vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array1 - An ARRAY of FLOAT values representing the first vector.
+ * array2 - An ARRAY of FLOAT values representing the second vector. It must have the
+ same dimension as array1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
@@ -101,6 +107,12 @@ case class VectorCosineSimilarity(left: Expression, right: Expression)
_FUNC_(array1, array2) - Returns the inner product (dot product) between two float vectors.
The vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array1 - An ARRAY of FLOAT values representing the first vector.
+ * array2 - An ARRAY of FLOAT values representing the second vector. It must have the
+ same dimension as array1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
@@ -159,6 +171,12 @@ case class VectorInnerProduct(left: Expression, right: Expression)
_FUNC_(array1, array2) - Returns the Euclidean (L2) distance between two float vectors.
The vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array1 - An ARRAY of FLOAT values representing the first vector.
+ * array2 - An ARRAY of FLOAT values representing the second vector. It must have the
+ same dimension as array1.
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(1.0F, 2.0F, 3.0F), array(4.0F, 5.0F, 6.0F));
@@ -218,6 +236,12 @@ case class VectorL2Distance(left: Expression, right: Expression)
Degree defaults to 2.0 (Euclidean norm) if unspecified. Supported values: 1.0 (L1 norm),
2.0 (L2 norm), float('inf') (infinity norm).
""",
+ arguments = """
+ Arguments:
+ * vector - An ARRAY of FLOAT values representing the vector.
+ * degree - A FLOAT specifying the norm degree. Defaults to 2.0 when omitted. Supported
+ values are 1.0 (L1 norm), 2.0 (L2 norm), and float('inf') (infinity norm).
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(3.0F, 4.0F), 2.0F);
@@ -284,6 +308,13 @@ case class VectorNorm(vector: Expression, degree: Expression)
Degree defaults to 2.0 (Euclidean norm) if unspecified. Supported values: 1.0 (L1 norm),
2.0 (L2 norm), float('inf') (infinity norm).
""",
+ arguments = """
+ Arguments:
+ * vector - An ARRAY of FLOAT values representing the vector to normalize.
+ * degree - A FLOAT specifying the norm degree used for normalization. Defaults to 2.0
+ when omitted. Supported values are 1.0 (L1 norm), 2.0 (L2 norm), and float('inf')
+ (infinity norm).
+ """,
examples = """
Examples:
> SELECT _FUNC_(array(3.0F, 4.0F), 2.0F);
@@ -555,6 +586,11 @@ trait VectorAggregateBase extends ImperativeAggregate
_FUNC_(array) - Returns the element-wise mean of float vectors in a group.
All vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array - An ARRAY of FLOAT values representing a vector. All vectors aggregated in the
+ group must have the same dimension.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (array(1.0F, 2.0F)), (array(3.0F, 4.0F)) AS tab(col);
@@ -635,6 +671,11 @@ case class VectorAvg(
_FUNC_(array) - Returns the element-wise sum of float vectors in a group.
All vectors must have the same dimension.
""",
+ arguments = """
+ Arguments:
+ * array - An ARRAY of FLOAT values representing a vector. All vectors aggregated in the
+ group must have the same dimension.
+ """,
examples = """
Examples:
> SELECT _FUNC_(col) FROM VALUES (array(1.0F, 2.0F)), (array(3.0F, 4.0F)) AS tab(col);
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
index 5a91d3720375..f75536c5475a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala
@@ -38,6 +38,13 @@ import org.apache.spark.unsafe.types.UTF8String
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(xmlStr, schema[, options]) - Returns a struct value with the given `xmlStr` and `schema`.",
+ arguments = """
+ Arguments:
+ * xmlStr - A string expression containing a single XML record to parse.
+ * schema - The schema of the output struct, as a DDL-formatted string or a schema
+ expression.
+ * options - Optional. A map of key-value pairs controlling how the XML is parsed.
+ """,
examples = """
Examples:
> SELECT _FUNC_('<p><a>1</a><b>0.8</b></p>', 'a INT, b DOUBLE');
@@ -139,6 +146,11 @@ case class XmlToStructs(
*/
@ExpressionDescription(
usage = "_FUNC_(xml[, options]) - Returns schema in the DDL format of XML string.",
+ arguments = """
+ Arguments:
+ * xml - A foldable string expression containing an XML record whose schema is inferred.
+ * options - Optional. A map of key-value pairs controlling how the XML is parsed.
+ """,
examples = """
Examples:
> SELECT _FUNC_('<p><a>1</a></p>');
@@ -223,6 +235,11 @@ case class SchemaOfXml(
// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(expr[, options]) - Returns a XML string with a given struct value",
+ arguments = """
+ Arguments:
+ * expr - A struct-valued expression to convert to an XML string.
+ * options - Optional. A map of key-value pairs controlling how the XML is generated.
+ """,
examples = """
Examples:
> SELECT _FUNC_(named_struct('a', 1, 'b', 2));
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
index e38c9e5f20b8..e61299161fdf 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/expressions/ExpressionInfoSuite.scala
@@ -174,11 +174,39 @@ class ExpressionInfoSuite extends SharedSparkSession {
}
test("SPARK-32870: Default expressions in FunctionRegistry should have their " +
- "usage, examples, since, and group filled") {
+ "usage, examples, arguments, since, and group filled") {
val ignoreSet = Set(
// Cast aliases do not need examples
"org.apache.spark.sql.catalyst.expressions.Cast")
+ // Functions that take no arguments are exempt from the `arguments` documentation
+ // requirement, since there is nothing to describe.
+ val noArgumentsSet = Set(
+ "org.apache.spark.sql.catalyst.expressions.Collations",
+ "org.apache.spark.sql.catalyst.expressions.CumeDist",
+ "org.apache.spark.sql.catalyst.expressions.CurDateExpressionBuilder",
+ "org.apache.spark.sql.catalyst.expressions.CurrentCatalog",
+ "org.apache.spark.sql.catalyst.expressions.CurrentDatabase",
+ "org.apache.spark.sql.catalyst.expressions.CurrentDate",
+ "org.apache.spark.sql.catalyst.expressions.CurrentPath",
+ "org.apache.spark.sql.catalyst.expressions.CurrentTimeZone",
+ "org.apache.spark.sql.catalyst.expressions.CurrentTimestamp",
+ "org.apache.spark.sql.catalyst.expressions.CurrentUser",
+ "org.apache.spark.sql.catalyst.expressions.EulerNumber",
+ "org.apache.spark.sql.catalyst.expressions.InputFileBlockLength",
+ "org.apache.spark.sql.catalyst.expressions.InputFileBlockStart",
+ "org.apache.spark.sql.catalyst.expressions.InputFileName",
+ "org.apache.spark.sql.catalyst.expressions.LocalTimestamp",
+ "org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID",
+ "org.apache.spark.sql.catalyst.expressions.Now",
+ "org.apache.spark.sql.catalyst.expressions.Pi",
+ "org.apache.spark.sql.catalyst.plans.logical.PythonWorkerLogs",
+ "org.apache.spark.sql.catalyst.expressions.RowNumber",
+ "org.apache.spark.sql.catalyst.expressions.SQLKeywords",
+ "org.apache.spark.sql.catalyst.expressions.SparkPartitionID",
+ "org.apache.spark.sql.catalyst.expressions.SparkVersion",
+ "org.apache.spark.sql.catalyst.expressions.Uuid")
+
spark.sessionState.functionRegistry.listFunction().foreach { funcId =>
val info = spark.sessionState.catalog.lookupFunctionInfo(funcId)
if (!ignoreSet.contains(info.getClassName)) {
@@ -189,6 +217,9 @@ class ExpressionInfoSuite extends SharedSparkSession {
assert(info.getSince.matches("[0-9]+\\.[0-9]+\\.[0-9]+"))
assert(info.getGroup.nonEmpty)
+ if (!noArgumentsSet.contains(info.getClassName)) {
+ assert(info.getArguments.nonEmpty)
+ }
if (info.getArguments.nonEmpty) {
assert(info.getArguments.startsWith("\n Arguments:\n"))
assert(info.getArguments.endsWith("\n "))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org