You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sa...@apache.org on 2026/10/16 06:57:23 UTC
(spark) branch master updated: [SPARK-60251][SQL][DOC] Document vector functions in the SQL built-in function reference
This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 95f8852bfa30 [SPARK-60251][SQL][DOC] Document vector functions in the SQL built-in function reference
95f8852bfa30 is described below
commit 95f8852bfa3023c90ff11c8c2711a1098a1aab27
Author: Kousuke Saruta <sa...@apache.org>
AuthorDate: Fri Oct 16 15:57:10 2026 +0900
[SPARK-60251][SQL][DOC] Document vector functions in the SQL built-in function reference
### What changes were proposed in this pull request?
This PR documents the vector functions (`vector_cosine_similarity`, `vector_inner_product`, `vector_l2_distance`, `vector_norm`, `vector_normalize`, `vector_avg`, `vector_sum`) in the SQL built-in function reference.
Three changes are made:
1. `sql/gen-sql-functions-docs.py`: add `vector_funcs` to the `groups` allowlist so the generator emits `generated-vector-funcs-table.html` and `generated-vector-funcs-examples.html` from the functions' `ExpressionDescription` (usage/examples/group).
2. `docs/sql-ref-functions-builtin.md`: add a "Vector Functions" section that includes the two generated HTML files, following the same pattern as the other function groups (e.g. "Geospatial ST Functions").
3. `docs/sql-ref-functions.md`: add a "Vector Functions" entry to the built-in function list so users can actually navigate to the new section. The entry is listed under both "Scalar Functions" and "Aggregate-like Functions", because the section documents both scalar functions and the aggregates `vector_avg` / `vector_sum` (which stay in the `vector_funcs` group rather than `agg_funcs`, so they do not appear in the Aggregate Functions table). Without these entries there is no link fro [...]
The vector functions themselves already exist as SQL built-ins (added in SPARK-55030, SPARK-55593 and SPARK-55031) with `group = "vector_funcs"`, and their Scala/Python bindings were added in SPARK-57284 (#56322). However, the group was never registered in the documentation generator, so these functions were missing from the built-in function reference docs and unreachable from the Functions reference page.
### Why are the changes needed?
The vector functions are user-facing SQL built-in functions but were not listed in the Built-in Functions reference page, unlike the other function groups, and there was no link to reach them from the Functions reference page. This closes that documentation gap so users can discover them and see their signatures and examples.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change (the functions already exist and behave as before).
### How was this patch tested?
Built the SQL function docs and the docs site locally and verified the rendered output:
- Ran `sql/gen-sql-functions-docs.py` (via `spark-submit`) and confirmed it produced `docs/generated-vector-funcs-table.html` (all 7 functions with descriptions) and `docs/generated-vector-funcs-examples.html` (each example SQL executed with correct results, e.g. `vector_cosine_similarity(array(1.0F,2.0F,3.0F), array(4.0F,5.0F,6.0F))` = `0.97463185`, `vector_l2_distance(...)` = `5.196152`, `vector_norm(array(3.0F,4.0F), float('inf'))` = `4.0`, `vector_normalize(array(3.0F,4.0F), 2.0F) [...]
- Ran `SKIP_API=1 bundle exec jekyll build` and confirmed:
- `_site/sql-ref-functions-builtin.html` renders a "Vector Functions" section with the function table followed by the examples, consistent with the other function groups.
- `_site/sql-ref-functions.html` lists "Vector Functions" under both "Scalar Functions" (between URL Functions and Misc Functions) and "Aggregate-like Functions", and both links resolve to `sql-ref-functions-builtin.html#vector-functions`.
* docs/sql-ref-functions-builtin.html
<img width="887" height="693" alt="vector-func-doc0" src="https://github.com/user-attachments/assets/5d01640d-b34f-4f87-b003-f3260e2a0ff8" />
* sql-ref-functions.html
<img width="852" height="743" alt="vector-func-doc1" src="https://github.com/user-attachments/assets/8238b54a-b92f-449d-a674-29d2e767acac" />
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI / Claude
Closes #59478 from sarutak/vector-funcs-doc.
Authored-by: Kousuke Saruta <sa...@apache.org>
Signed-off-by: Kousuke Saruta <sa...@apache.org>
---
docs/sql-ref-functions-builtin.md | 5 +++++
docs/sql-ref-functions.md | 2 ++
sql/gen-sql-functions-docs.py | 2 +-
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/docs/sql-ref-functions-builtin.md b/docs/sql-ref-functions-builtin.md
index b7c90c8a846a..1fe2dd6f97ef 100644
--- a/docs/sql-ref-functions-builtin.md
+++ b/docs/sql-ref-functions-builtin.md
@@ -140,3 +140,8 @@ user-defined function that may share the same name.
{% include_api_gen generated-st-funcs-table.html %}
#### Examples
{% include_api_gen generated-st-funcs-examples.html %}
+
+### Vector Functions
+{% include_api_gen generated-vector-funcs-table.html %}
+#### Examples
+{% include_api_gen generated-vector-funcs-examples.html %}
diff --git a/docs/sql-ref-functions.md b/docs/sql-ref-functions.md
index 2b50dbe862e0..9663b763f829 100644
--- a/docs/sql-ref-functions.md
+++ b/docs/sql-ref-functions.md
@@ -44,12 +44,14 @@ This subsection presents the usages and descriptions of these functions.
* [JSON Functions](sql-ref-functions-builtin.html#json-functions)
* [XML Functions](sql-ref-functions-builtin.html#xml-functions)
* [URL Functions](sql-ref-functions-builtin.html#url-functions)
+ * [Vector Functions](sql-ref-functions-builtin.html#vector-functions)
* [Misc Functions](sql-ref-functions-builtin.html#misc-functions)
#### Aggregate-like Functions
* [Aggregate Functions](sql-ref-functions-builtin.html#aggregate-functions)
* [Window Functions](sql-ref-functions-builtin.html#window-functions)
* [Sketch-Based Approximate Functions](sql-ref-sketch-aggregates.html)
+ * [Vector Functions](sql-ref-functions-builtin.html#vector-functions)
#### Generator Functions
* [Generator Functions](sql-ref-functions-builtin.html#generator-functions)
diff --git a/sql/gen-sql-functions-docs.py b/sql/gen-sql-functions-docs.py
index 0fe1d4c69d4f..ea8700eaa2be 100644
--- a/sql/gen-sql-functions-docs.py
+++ b/sql/gen-sql-functions-docs.py
@@ -35,7 +35,7 @@ groups = {
"xml_funcs", "lambda_funcs", "collection_funcs",
"url_funcs", "hash_funcs", "struct_funcs",
"table_funcs", "variant_funcs", "protobuf_funcs", "sketch_funcs",
- "st_funcs"
+ "st_funcs", "vector_funcs"
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org