You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2022/06/27 13:51:36 UTC
[spark] branch master updated: [SPARK-39577][SQL][DOCS] Add SQL reference for built-in functions
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 250cb912215 [SPARK-39577][SQL][DOCS] Add SQL reference for built-in functions
250cb912215 is described below
commit 250cb912215e548b965aa2d1a27affe9f924cac7
Author: Jiaan Geng <be...@163.com>
AuthorDate: Mon Jun 27 21:51:21 2022 +0800
[SPARK-39577][SQL][DOCS] Add SQL reference for built-in functions
### What changes were proposed in this pull request?
Currently, Spark SQL reference missing many functions.
Users cannot find the needed functions.
### Why are the changes needed?
Add SQL reference for built-in functions
### Does this PR introduce _any_ user-facing change?
'Yes'.
Users can find needed functions in SQL reference.
Before this PR, the built-in functions show below.
After this PR, the built-in functions show below.
The part of Mathematical Functions show below.
The part of String Functions show below.
The part of Bitwise Functions show below.
The part of Conversion Functions show below.
The part of Conditional Functions show below.
The part of Predicate Functions show below.
The part of Csv Functions show below.
The part of Misc Functions show below.
The part of Generator Functions show below.
### How was this patch tested?
N/A
Closes #36976 from beliefer/SPARK-39577.
Authored-by: Jiaan Geng <be...@163.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
docs/sql-ref-functions-builtin.md | 89 +++++++++++++++++++++++++++++++++++++++
docs/sql-ref-functions.md | 11 +++++
sql/gen-sql-functions-docs.py | 12 +++++-
3 files changed, 111 insertions(+), 1 deletion(-)
diff --git a/docs/sql-ref-functions-builtin.md b/docs/sql-ref-functions-builtin.md
index 08e620b0f4f..c28905c3fdc 100644
--- a/docs/sql-ref-functions-builtin.md
+++ b/docs/sql-ref-functions-builtin.md
@@ -77,3 +77,92 @@ license: |
{% endif %}
{% endfor %}
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-math-funcs-table.html' %}
+### Mathematical Functions
+{% include_relative generated-math-funcs-table.html %}
+#### Examples
+{% include_relative generated-math-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-string-funcs-table.html' %}
+### String Functions
+{% include_relative generated-string-funcs-table.html %}
+#### Examples
+{% include_relative generated-string-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-conditional-funcs-table.html' %}
+### Conditional Functions
+{% include_relative generated-conditional-funcs-table.html %}
+#### Examples
+{% include_relative generated-conditional-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-bitwise-funcs-table.html' %}
+### Bitwise Functions
+{% include_relative generated-bitwise-funcs-table.html %}
+#### Examples
+{% include_relative generated-bitwise-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-conversion-funcs-table.html' %}
+### Conversion Functions
+{% include_relative generated-conversion-funcs-table.html %}
+#### Examples
+{% include_relative generated-conversion-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-predicate-funcs-table.html' %}
+### Predicate Functions
+{% include_relative generated-predicate-funcs-table.html %}
+#### Examples
+{% include_relative generated-predicate-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-csv-funcs-table.html' %}
+### Csv Functions
+{% include_relative generated-csv-funcs-table.html %}
+#### Examples
+{% include_relative generated-csv-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-misc-funcs-table.html' %}
+### Misc Functions
+{% include_relative generated-misc-funcs-table.html %}
+#### Examples
+{% include_relative generated-misc-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
+
+{% for static_file in site.static_files %}
+ {% if static_file.name == 'generated-generator-funcs-table.html' %}
+### Generator Functions
+{% include_relative generated-generator-funcs-table.html %}
+#### Examples
+{% include_relative generated-generator-funcs-examples.html %}
+ {% break %}
+ {% endif %}
+{% endfor %}
diff --git a/docs/sql-ref-functions.md b/docs/sql-ref-functions.md
index 67951a9695f..58be9e7e619 100644
--- a/docs/sql-ref-functions.md
+++ b/docs/sql-ref-functions.md
@@ -32,11 +32,22 @@ This subsection presents the usages and descriptions of these functions.
* [Map Functions](sql-ref-functions-builtin.html#map-functions)
* [Date and Timestamp Functions](sql-ref-functions-builtin.html#date-and-timestamp-functions)
* [JSON Functions](sql-ref-functions-builtin.html#json-functions)
+ * [Mathematical Functions](sql-ref-functions-builtin.html#mathematical-functions)
+ * [String Functions](sql-ref-functions-builtin.html#string-functions)
+ * [Bitwise Functions](sql-ref-functions-builtin.html#bitwise-functions)
+ * [Conversion Functions](sql-ref-functions-builtin.html#conversion-functions)
+ * [Conditional Functions](sql-ref-functions-builtin.html#conditional-functions)
+ * [Predicate Functions](sql-ref-functions-builtin.html#predicate-functions)
+ * [Csv Functions](sql-ref-functions-builtin.html#csv-functions)
+ * [Misc Functions](sql-ref-functions-builtin.html#misc-functions)
#### Aggregate-like Functions
* [Aggregate Functions](sql-ref-functions-builtin.html#aggregate-functions)
* [Window Functions](sql-ref-functions-builtin.html#window-functions)
+#### Generator Functions
+* [Generator Functions](sql-ref-functions-builtin.html#generator-functions)
+
### UDFs (User-Defined Functions)
User-Defined Functions (UDFs) are a feature of Spark SQL that allows users to define their own functions when the system's built-in functions are not enough to perform the desired task. To use UDFs in Spark SQL, users must first define the function, then register the function with Spark, and finally call the registered function. The User-Defined Functions can act on a single row or act on multiple rows at once. Spark SQL also supports integration of existing Hive implementations of UDFs, [...]
diff --git a/sql/gen-sql-functions-docs.py b/sql/gen-sql-functions-docs.py
index c07734e2730..8d18b06768c 100644
--- a/sql/gen-sql-functions-docs.py
+++ b/sql/gen-sql-functions-docs.py
@@ -31,6 +31,9 @@ ExpressionInfo = namedtuple("ExpressionInfo", "name usage examples group")
groups = {
"agg_funcs", "array_funcs", "datetime_funcs",
"json_funcs", "map_funcs", "window_funcs",
+ "math_funcs", "conditional_funcs", "generator_funcs",
+ "predicate_funcs", "string_funcs", "misc_funcs",
+ "bitwise_funcs", "conversion_funcs", "csv_funcs",
}
@@ -45,6 +48,8 @@ def _list_grouped_function_infos(jvm):
for jinfo in filter(lambda x: x.getGroup() in groups, jinfos):
name = jinfo.getName()
+ if (name == "raise_error"):
+ continue
usage = jinfo.getUsage()
usage = usage.replace("_FUNC_", name) if usage is not None else usage
infos.append(ExpressionInfo(
@@ -108,7 +113,12 @@ def _make_pretty_usage(infos):
# Expected formats are as follows;
# - `_FUNC_(...) - description`, or
# - `_FUNC_ - description`
- usages = iter(re.split(r"(%s.*) - " % info.name, info.usage.strip())[1:])
+ func_name = info.name
+ if (info.name == "*" or info.name == "+"):
+ func_name = "\\" + func_name
+ elif (info.name == "when"):
+ func_name = "CASE WHEN"
+ usages = iter(re.split(r"(.*%s.*) - " % func_name, info.usage.strip())[1:])
for (sig, description) in zip(usages, usages):
result.append(" <tr>")
result.append(" <td>%s</td>" % sig)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org