You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/04 16:26:29 UTC
(datafusion-python) branch asf-staging updated: Publish built docs triggered by a2ddb5531ac8443a4593101f8a801cb4a0088c01
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new ff662638 Publish built docs triggered by a2ddb5531ac8443a4593101f8a801cb4a0088c01
ff662638 is described below
commit ff662638911a0db3f60a569ab14af7380a8f7465
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Sun Oct 4 16:26:23 2026 +0000
Publish built docs triggered by a2ddb5531ac8443a4593101f8a801cb4a0088c01
---
_sources/autoapi/datafusion/catalog/index.rst.txt | 17 +
_sources/autoapi/datafusion/context/index.rst.txt | 6 +-
.../autoapi/datafusion/dataframe/index.rst.txt | 120 ++-
_sources/autoapi/datafusion/expr/index.rst.txt | 35 +-
.../autoapi/datafusion/extensions/index.rst.txt | 2 +-
.../autoapi/datafusion/functions/index.rst.txt | 607 ++++++++++--
.../datafusion/functions/spark/index.rst.txt | 236 ++++-
_sources/autoapi/datafusion/index.rst.txt | 35 +-
.../autoapi/datafusion/user_defined/index.rst.txt | 31 +-
.../common-operations/aggregations.md.txt | 3 +-
.../common-operations/expressions.md.txt | 5 +-
.../user-guide/common-operations/functions.md.txt | 35 +
.../user-guide/common-operations/windows.md.txt | 71 +-
_sources/user-guide/upgrade-guides.md.txt | 173 ++++
autoapi/datafusion/catalog/index.html | 20 +
autoapi/datafusion/context/index.html | 8 +-
autoapi/datafusion/dataframe/index.html | 140 ++-
autoapi/datafusion/expr/index.html | 33 +-
autoapi/datafusion/extensions/index.html | 2 +-
autoapi/datafusion/functions/index.html | 1005 +++++++++++++++-----
autoapi/datafusion/functions/spark/index.html | 406 +++++++-
autoapi/datafusion/index.html | 33 +-
autoapi/datafusion/user_defined/index.html | 36 +-
genindex.html | 152 ++-
objects.inv | Bin 12981 -> 13365 bytes
searchindex.js | 2 +-
user-guide/common-operations/aggregations.html | 3 +-
user-guide/common-operations/basic-info.html | 46 +-
user-guide/common-operations/expressions.html | 5 +-
user-guide/common-operations/functions.html | 51 +-
user-guide/common-operations/index.html | 2 +
user-guide/common-operations/udf-and-udfa.html | 4 +-
user-guide/common-operations/windows.html | 68 +-
user-guide/upgrade-guides.html | 156 +++
34 files changed, 3026 insertions(+), 522 deletions(-)
diff --git a/_sources/autoapi/datafusion/catalog/index.rst.txt b/_sources/autoapi/datafusion/catalog/index.rst.txt
index dcd21725..939d195a 100644
--- a/_sources/autoapi/datafusion/catalog/index.rst.txt
+++ b/_sources/autoapi/datafusion/catalog/index.rst.txt
@@ -21,6 +21,7 @@ Classes
datafusion.catalog.Schema
datafusion.catalog.SchemaProvider
datafusion.catalog.Table
+ datafusion.catalog.TableProviderFactory
Module Contents
@@ -376,3 +377,19 @@ Module Contents
Returns the schema associated with this table.
+.. py:class:: TableProviderFactory
+
+ Bases: :py:obj:`abc.ABC`
+
+
+ Abstract class for defining a Python based Table Provider Factory.
+
+
+ .. py:method:: create(cmd: datafusion.expr.CreateExternalTable) -> Table
+ :abstractmethod:
+
+
+ Create a table using the :class:`CreateExternalTable`.
+
+
+
diff --git a/_sources/autoapi/datafusion/context/index.rst.txt b/_sources/autoapi/datafusion/context/index.rst.txt
index dcffc800..25b88317 100644
--- a/_sources/autoapi/datafusion/context/index.rst.txt
+++ b/_sources/autoapi/datafusion/context/index.rst.txt
@@ -1653,7 +1653,7 @@ Module Contents
- .. py:method:: set_query_planner(planner: datafusion.extensions.QueryPlannerExportable | _typeshed.CapsuleType) -> None
+ .. py:method:: set_query_planner(planner: datafusion.extensions.QueryPlannerExportable | types.CapsuleType) -> None
Install a custom query planner on this session.
@@ -1998,7 +1998,7 @@ Module Contents
- .. py:method:: with_logical_extension_codec(codec: datafusion.user_defined.LogicalExtensionCodecExportable | _typeshed.CapsuleType, codec_id: str | None = None) -> SessionContext
+ .. py:method:: with_logical_extension_codec(codec: datafusion.user_defined.LogicalExtensionCodecExportable | types.CapsuleType, codec_id: str | None = None) -> SessionContext
Create a new session context with an additional logical codec.
@@ -2061,7 +2061,7 @@ Module Contents
- .. py:method:: with_physical_extension_codec(codec: datafusion.user_defined.PhysicalExtensionCodecExportable | _typeshed.CapsuleType, codec_id: str | None = None) -> SessionContext
+ .. py:method:: with_physical_extension_codec(codec: datafusion.user_defined.PhysicalExtensionCodecExportable | types.CapsuleType, codec_id: str | None = None) -> SessionContext
Create a new session context with an additional physical codec.
diff --git a/_sources/autoapi/datafusion/dataframe/index.rst.txt b/_sources/autoapi/datafusion/dataframe/index.rst.txt
index bdc9fd78..3f737cd5 100644
--- a/_sources/autoapi/datafusion/dataframe/index.rst.txt
+++ b/_sources/autoapi/datafusion/dataframe/index.rst.txt
@@ -43,7 +43,9 @@ Classes
datafusion.dataframe.Compression
datafusion.dataframe.DataFrame
datafusion.dataframe.DataFrameWriteOptions
+ datafusion.dataframe.ExplainAnalyzeLevel
datafusion.dataframe.ExplainFormat
+ datafusion.dataframe.ExplainMetricCategory
datafusion.dataframe.InsertOp
datafusion.dataframe.ParquetColumnOptions
datafusion.dataframe.ParquetWriterOptions
@@ -524,7 +526,7 @@ Module Contents
- .. py:method:: explain(verbose: bool = False, analyze: bool = False, format: ExplainFormat | None = None) -> None
+ .. py:method:: explain(verbose: bool = False, analyze: bool = False, format: ExplainFormat | None = None, show_statistics: bool | None = None, analyze_level: ExplainAnalyzeLevel | None = None, analyze_categories: collections.abc.Iterable[ExplainMetricCategory] | None = None) -> None
Print an explanation of the DataFrame's plan so far.
@@ -534,6 +536,17 @@ Module Contents
:param analyze: If ``True``, the plan will run and metrics reported.
:param format: Output format for the plan. Defaults to
:py:attr:`ExplainFormat.INDENT`.
+ :param show_statistics: If ``True``, include each operator's statistics.
+ ``None`` uses the ``datafusion.explain.show_statistics``
+ setting.
+ :param analyze_level: Which metrics to report with ``analyze``. ``None``
+ uses the ``datafusion.explain.analyze_level`` setting.
+ :param analyze_categories: Report only metrics in these categories with
+ ``analyze``; an empty iterable reports none. ``None`` uses the
+ ``datafusion.explain.analyze_categories`` setting.
+
+ :raises ValueError: If ``show_statistics`` is set with ``analyze``, or
+ ``analyze_level`` or ``analyze_categories`` is set without it.
.. rubric:: Examples
@@ -548,6 +561,45 @@ Module Contents
>>> df.explain(analyze=True) # doctest: +SKIP
+ Show only row-count metrics:
+
+ >>> from datafusion.dataframe import ExplainMetricCategory
+ >>> df.explain(
+ ... analyze=True, analyze_categories=[ExplainMetricCategory.ROWS]
+ ... ) # doctest: +SKIP
+
+
+
+ .. py:method:: fill_nan(value: float, subset: list[str] | None = None) -> DataFrame
+
+ Fill NaN values in floating-point columns with a value.
+
+ Only floating-point columns are changed; others are kept unchanged, as is
+ any column ``value`` cannot be cast to. NaN is distinct from null, which
+ :py:meth:`fill_null` handles. Fails on a DataFrame with an uppercase or
+ dotted column name; see :ref:`fill_column_names`.
+
+ :param value: Value to replace NaN with. Will be cast to match column type.
+ :param subset: Optional list of column names to fill. If None, fills all
+ floating-point columns; an empty list fills none.
+
+ :returns: DataFrame with NaN values replaced.
+
+ .. rubric:: Examples
+
+ >>> from datafusion import SessionContext
+ >>> ctx = SessionContext()
+ >>> nan = float("nan")
+ >>> df = ctx.from_pydict({"a": [1.0, nan, None], "b": [nan, 2.0, 3.0]})
+ >>> df.fill_nan(0.0).to_pydict()
+ {'a': [1.0, 0.0, None], 'b': [0.0, 2.0, 3.0]}
+
+ >>> df.fill_nan(0.0, subset=["a"]).collect_column("b")[0].as_py()
+ nan
+
+ >>> df.fill_nan(0.0, subset=[]).collect_column("b")[0].as_py()
+ nan
+
.. py:method:: fill_null(value: Any, subset: list[str] | None = None) -> DataFrame
@@ -555,7 +607,8 @@ Module Contents
Fill null values in specified columns with a value.
:param value: Value to replace nulls with. Will be cast to match column type.
- :param subset: Optional list of column names to fill. If None, fills all columns.
+ :param subset: Optional list of column names to fill. If None, fills all columns;
+ an empty list fills none.
:returns: DataFrame with null values replaced where type casting is possible
@@ -568,11 +621,16 @@ Module Contents
>>> filled.sort(col("a")).collect()[0].column("a").to_pylist()
[0, 1, 3]
+ >>> df.fill_null(0, subset=[]).to_pydict()
+ {'a': [1, None, 3], 'b': [None, 5, 6]}
+
.. rubric:: Notes
- Only fills nulls in columns where the value can be cast to the column type
- For columns where casting fails, the original column is kept unchanged
- For columns not in subset, the original column is kept unchanged
+ - Fails on a DataFrame with an uppercase or dotted column name; see
+ :ref:`fill_column_names`
@@ -1345,6 +1403,28 @@ Module Contents
.. py:attribute:: _raw_write_options
+.. py:class:: ExplainAnalyzeLevel
+
+ Bases: :py:obj:`enum.Enum`
+
+
+ Which metrics :py:meth:`DataFrame.explain` reports when ``analyze=True``.
+
+
+ .. py:attribute:: DEV
+ :value: 'dev'
+
+
+ All metrics, including those for deep operator-level introspection.
+
+
+ .. py:attribute:: SUMMARY
+ :value: 'summary'
+
+
+ Common metrics for finding which operator is slow.
+
+
.. py:class:: ExplainFormat
Bases: :py:obj:`enum.Enum`
@@ -1383,6 +1463,42 @@ Module Contents
Tree-style visual format with box-drawing characters.
+.. py:class:: ExplainMetricCategory
+
+ Bases: :py:obj:`enum.Enum`
+
+
+ Category of metric reported by :py:meth:`DataFrame.explain` with ``analyze``.
+
+
+ .. py:attribute:: BYTES
+ :value: 'bytes'
+
+
+ Byte sizes, such as ``output_bytes``.
+
+
+ .. py:attribute:: ROWS
+ :value: 'rows'
+
+
+ Row counts, such as ``output_rows``.
+
+
+ .. py:attribute:: TIMING
+ :value: 'timing'
+
+
+ Elapsed times, such as ``elapsed_compute``.
+
+
+ .. py:attribute:: UNCATEGORIZED
+ :value: 'uncategorized'
+
+
+ Metrics that declare no category.
+
+
.. py:class:: InsertOp
Bases: :py:obj:`enum.Enum`
diff --git a/_sources/autoapi/datafusion/expr/index.rst.txt b/_sources/autoapi/datafusion/expr/index.rst.txt
index 88b10a19..6a00701d 100644
--- a/_sources/autoapi/datafusion/expr/index.rst.txt
+++ b/_sources/autoapi/datafusion/expr/index.rst.txt
@@ -631,8 +631,9 @@ Module Contents
Only evaluate distinct values for an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on a window
+ function unless that is an aggregate used as one. If used on any other type of
+ expression, an error will be generated when ``build()`` is called.
@@ -671,8 +672,9 @@ Module Contents
Filter an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on a window
+ function unless that is an aggregate used as one. If used on any other type of
+ expression, an error will be generated when ``build()`` is called.
@@ -904,12 +906,19 @@ Module Contents
.. py:method:: over(window: Window) -> Expr
- Turn an aggregate function into a window function.
+ Evaluate an aggregate or window function over a window.
- This function turns any aggregate function into a window function. With the
+ On an aggregate function this turns it into a window function. With the
exception of ``partition_by``, how each of the parameters is used is determined
by the underlying aggregate function.
+ On an aggregate, the ``null_treatment``, ``filter``, and ``distinct`` options
+ it was built with are kept, and an ``order_by`` raises; see
+ :ref:`aggregate_over_options`.
+
+ On a window function that already has a ``partition_by``, ``order_by``, or
+ ``window_frame``, this raises; see :ref:`window_function_chaining`.
+
:param window: Window definition
@@ -919,8 +928,10 @@ Module Contents
Set the partitioning for a window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on an
+ aggregate function that has not been turned into a window function with
+ :py:meth:`over`. If used on any other type of expression, an error will be
+ generated when ``build()`` is called.
@@ -1214,11 +1225,13 @@ Module Contents
.. py:method:: window_frame(window_frame: WindowFrame) -> ExprFuncBuilder
- Set the frame fora window function.
+ Set the frame for a window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on an
+ aggregate function that has not been turned into a window function with
+ :py:meth:`over`. If used on any other type of expression, an error will be
+ generated when ``build()`` is called.
diff --git a/_sources/autoapi/datafusion/extensions/index.rst.txt b/_sources/autoapi/datafusion/extensions/index.rst.txt
index 51d5dbb8..08745ee1 100644
--- a/_sources/autoapi/datafusion/extensions/index.rst.txt
+++ b/_sources/autoapi/datafusion/extensions/index.rst.txt
@@ -340,6 +340,6 @@ Module Contents
False
- .. py:method:: __datafusion_session_planner__(ctx: datafusion.context.SessionContext, fallback: _typeshed.CapsuleType) -> QueryPlannerExportable | _typeshed.CapsuleType | None
+ .. py:method:: __datafusion_session_planner__(ctx: datafusion.context.SessionContext, fallback: types.CapsuleType) -> QueryPlannerExportable | types.CapsuleType | None
diff --git a/_sources/autoapi/datafusion/functions/index.rst.txt b/_sources/autoapi/datafusion/functions/index.rst.txt
index 7fedb656..6115b7ad 100644
--- a/_sources/autoapi/datafusion/functions/index.rst.txt
+++ b/_sources/autoapi/datafusion/functions/index.rst.txt
@@ -56,15 +56,18 @@ Functions
datafusion.functions.acosh
datafusion.functions.alias
datafusion.functions.any_match
+ datafusion.functions.any_value
datafusion.functions.approx_distinct
datafusion.functions.approx_median
datafusion.functions.approx_percentile_cont
datafusion.functions.approx_percentile_cont_with_weight
datafusion.functions.array
+ datafusion.functions.array_add
datafusion.functions.array_agg
datafusion.functions.array_any_match
datafusion.functions.array_any_value
datafusion.functions.array_append
+ datafusion.functions.array_avg
datafusion.functions.array_cat
datafusion.functions.array_compact
datafusion.functions.array_concat
@@ -77,6 +80,7 @@ Functions
datafusion.functions.array_except
datafusion.functions.array_extract
datafusion.functions.array_filter
+ datafusion.functions.array_first
datafusion.functions.array_has
datafusion.functions.array_has_all
datafusion.functions.array_has_any
@@ -93,6 +97,7 @@ Functions
datafusion.functions.array_position
datafusion.functions.array_positions
datafusion.functions.array_prepend
+ datafusion.functions.array_product
datafusion.functions.array_push_back
datafusion.functions.array_push_front
datafusion.functions.array_remove
@@ -104,8 +109,11 @@ Functions
datafusion.functions.array_replace_n
datafusion.functions.array_resize
datafusion.functions.array_reverse
+ datafusion.functions.array_scale
datafusion.functions.array_slice
datafusion.functions.array_sort
+ datafusion.functions.array_subtract
+ datafusion.functions.array_sum
datafusion.functions.array_to_string
datafusion.functions.array_transform
datafusion.functions.array_union
@@ -175,6 +183,7 @@ Functions
datafusion.functions.exp
datafusion.functions.extract
datafusion.functions.factorial
+ datafusion.functions.file_row_index
datafusion.functions.find_in_set
datafusion.functions.first_value
datafusion.functions.flatten
@@ -190,6 +199,7 @@ Functions
datafusion.functions.in_list
datafusion.functions.initcap
datafusion.functions.inner_product
+ datafusion.functions.input_file_name
datafusion.functions.instr
datafusion.functions.is_nan
datafusion.functions.isnan
@@ -204,9 +214,11 @@ Functions
datafusion.functions.left
datafusion.functions.length
datafusion.functions.levenshtein
+ datafusion.functions.list_add
datafusion.functions.list_any_match
datafusion.functions.list_any_value
datafusion.functions.list_append
+ datafusion.functions.list_avg
datafusion.functions.list_cat
datafusion.functions.list_compact
datafusion.functions.list_concat
@@ -219,6 +231,7 @@ Functions
datafusion.functions.list_except
datafusion.functions.list_extract
datafusion.functions.list_filter
+ datafusion.functions.list_first
datafusion.functions.list_has
datafusion.functions.list_has_all
datafusion.functions.list_has_any
@@ -236,6 +249,7 @@ Functions
datafusion.functions.list_position
datafusion.functions.list_positions
datafusion.functions.list_prepend
+ datafusion.functions.list_product
datafusion.functions.list_push_back
datafusion.functions.list_push_front
datafusion.functions.list_remove
@@ -247,8 +261,11 @@ Functions
datafusion.functions.list_replace_n
datafusion.functions.list_resize
datafusion.functions.list_reverse
+ datafusion.functions.list_scale
datafusion.functions.list_slice
datafusion.functions.list_sort
+ datafusion.functions.list_subtract
+ datafusion.functions.list_sum
datafusion.functions.list_to_string
datafusion.functions.list_transform
datafusion.functions.list_union
@@ -293,6 +310,7 @@ Functions
datafusion.functions.power
datafusion.functions.quantile_cont
datafusion.functions.radians
+ datafusion.functions.rand
datafusion.functions.random
datafusion.functions.range
datafusion.functions.rank
@@ -340,6 +358,7 @@ Functions
datafusion.functions.substr
datafusion.functions.substr_index
datafusion.functions.substring
+ datafusion.functions.substring_index
datafusion.functions.sum
datafusion.functions.tan
datafusion.functions.tanh
@@ -452,6 +471,40 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_any_match`.
+.. py:function:: any_value(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+
+ Returns an arbitrary non-null value from each group.
+
+ Returns NULL if every value in the group is NULL. Which value is returned
+ is not specified and may differ between runs.
+
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the options ``order_by``, ``null_treatment``, and ``distinct``.
+
+ :param expression: Argument to pick a value from
+ :param filter: If provided, only consider rows for which the filter is True
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": [None, 7, None]})
+ >>> result = df.aggregate(
+ ... [], [dfn.functions.any_value(dfn.col("a")).alias("v")]
+ ... )
+ >>> result.collect_column("v")[0].as_py()
+ 7
+
+ >>> df = ctx.from_pydict({"a": [None, 7, 8], "b": [1, 2, 3]})
+ >>> result = df.aggregate(
+ ... [], [dfn.functions.any_value(
+ ... dfn.col("a"),
+ ... filter=dfn.col("b") > dfn.lit(2)
+ ... ).alias("v")]
+ ... )
+ >>> result.collect_column("v")[0].as_py()
+ 8
+
+
.. py:function:: approx_distinct(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the approximate number of distinct values.
@@ -460,7 +513,7 @@ Package Contents
will approximate the number of distinct entries. It may return significantly faster
than :py:func:`count` for some DataFrames.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: Values to check for distinct entries
@@ -493,7 +546,7 @@ Package Contents
This aggregate function is similar to :py:func:`median`, but it will only
approximate the median. It may return significantly faster for some DataFrames.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by`` and ``null_treatment``, and ``distinct``.
:param expression: Values to find the median for
@@ -533,8 +586,10 @@ Package Contents
compute the percentile. You can limit the number of bins used in this algorithm by
setting the ``num_centroids`` parameter.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``order_by``, ``null_treatment``, and ``distinct``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the options ``null_treatment`` and ``distinct``. A chained ``order_by`` sets only
+ the sort direction; the percentile is still computed over ``sort_expression``,
+ so pass the same expression.
:param sort_expression: Values for which to find the approximate percentile
:param percentile: This must be between 0.0 and 1.0, inclusive
@@ -569,8 +624,10 @@ Package Contents
This aggregate function is similar to :py:func:`approx_percentile_cont` except that
it uses the associated associated weights.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``order_by``, ``null_treatment``, and ``distinct``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the option ``null_treatment`` and rejects ``distinct``. A chained ``order_by``
+ sets only the sort direction; the percentile is still computed over
+ ``sort_expression``, so pass the same expression.
:param sort_expression: Values for which to find the approximate percentile
:param weight: Relative weight for each of the values in ``expression``
@@ -606,6 +663,26 @@ Package Contents
.. seealso:: This is an alias for :py:func:`make_array`.
+.. py:function:: array_add(array1: datafusion.expr.Expr, array2: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the element-wise sum of two numeric arrays of equal length.
+
+ A NULL element in either input produces a NULL at that position. Execution
+ fails if the arrays in a row have different lengths.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict(
+ ... {"a": [[1.0, 2.0, 3.0]], "b": [[10.0, 20.0, 30.0]]}
+ ... )
+ >>> result = df.select(
+ ... dfn.functions.array_add(dfn.col("a"), dfn.col("b")).alias("result")
+ ... )
+ >>> result.collect_column("result")[0].as_py()
+ [11.0, 22.0, 33.0]
+
+
.. py:function:: array_agg(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Aggregate values into an array.
@@ -614,7 +691,7 @@ Package Contents
consider :py:func:`array_sort` after aggregation.
[Issue Tracker](https://github.com/apache/datafusion/issues/12371)
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the option ``null_treatment``.
:param expression: Values to combine into an array
@@ -709,6 +786,24 @@ Package Contents
[1, 2, 3, 4]
+.. py:function:: array_avg(array: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the arithmetic mean of the elements of a numeric array.
+
+ NULL elements are skipped and excluded from the count. Returns NULL if the
+ array is NULL, empty, or contains only NULL elements.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": [[1.0, None, 3.0]]})
+ >>> result = df.select(
+ ... dfn.functions.array_avg(dfn.col("a")).alias("result")
+ ... )
+ >>> result.collect_column("result")[0].as_py()
+ 2.0
+
+
.. py:function:: array_cat(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
Concatenates the input arrays.
@@ -879,6 +974,36 @@ Package Contents
.. seealso:: :py:func:`array_transform`, :py:func:`array_any_match`, :py:func:`lambda_`.
+.. py:function:: array_first(array: datafusion.expr.Expr, predicate: datafusion.expr.Expr | collections.abc.Callable[Ellipsis, Any]) -> datafusion.expr.Expr
+
+ Return the first element of ``array`` for which ``predicate`` is ``True``.
+
+ ``predicate`` may be a Python callable, converted to a lambda
+ automatically, or an explicit lambda built with :py:func:`lambda_`. It must
+ return a boolean expression. Returns NULL if no element matches.
+
+ .. rubric:: Examples
+
+ Using a Python callable:
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": [[1, 2, 3, 4]]})
+ >>> df.select(
+ ... F.array_first(col("a"), lambda v: v > 2).alias("f")
+ ... ).collect_column("f")[0].as_py()
+ 3
+
+ Using an explicit lambda built with :py:func:`lambda_`:
+
+ >>> predicate = F.lambda_(["v"], F.lambda_var("v") > lit(2))
+ >>> df.select(
+ ... F.array_first(col("a"), predicate).alias("f")
+ ... ).collect_column("f")[0].as_py()
+ 3
+
+ .. seealso:: :py:func:`array_filter`, :py:func:`array_any_match`, :py:func:`lambda_`.
+
+
.. py:function:: array_has(first_array: datafusion.expr.Expr, second_array: datafusion.expr.Expr) -> datafusion.expr.Expr
Returns true if the element appears in the first array, otherwise false.
@@ -953,7 +1078,7 @@ Package Contents
[2, 3]
-.. py:function:: array_join(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str) -> datafusion.expr.Expr
+.. py:function:: array_join(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, null_string: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
Converts each element to its text representation.
@@ -1133,6 +1258,24 @@ Package Contents
[0, 1, 2]
+.. py:function:: array_product(array: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the product of the elements of a numeric array.
+
+ NULL elements are skipped. Returns NULL if the array is NULL, empty, or
+ contains only NULL elements.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": [[2.0, None, 3.0]]})
+ >>> result = df.select(
+ ... dfn.functions.array_product(dfn.col("a")).alias("result")
+ ... )
+ >>> result.collect_column("result")[0].as_py()
+ 6.0
+
+
.. py:function:: array_push_back(array: datafusion.expr.Expr, element: datafusion.expr.Expr) -> datafusion.expr.Expr
Appends an element to the end of an array.
@@ -1287,6 +1430,24 @@ Package Contents
[3, 2, 1]
+.. py:function:: array_scale(array: datafusion.expr.Expr, scalar: datafusion.expr.Expr | float) -> datafusion.expr.Expr
+
+ Multiplies each element of a numeric array by ``scalar``.
+
+ A NULL element produces a NULL at that position. Returns NULL if ``scalar``
+ is NULL.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": [[1.0, 2.0, 3.0]]})
+ >>> result = df.select(
+ ... dfn.functions.array_scale(dfn.col("a"), 2.0).alias("result")
+ ... )
+ >>> result.collect_column("result")[0].as_py()
+ [2.0, 4.0, 6.0]
+
+
.. py:function:: array_slice(array: datafusion.expr.Expr, begin: datafusion.expr.Expr | int, end: datafusion.expr.Expr | int, stride: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
Returns a slice of the array.
@@ -1338,18 +1499,69 @@ Package Contents
[None, 3, 1]
-.. py:function:: array_to_string(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str) -> datafusion.expr.Expr
+.. py:function:: array_subtract(array1: datafusion.expr.Expr, array2: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the element-wise difference of two numeric arrays of equal length.
+
+ Computes ``array1[i] - array2[i]``. A NULL element in either input produces
+ a NULL at that position. Execution fails if the arrays in a row have
+ different lengths.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict(
+ ... {"a": [[10.0, 20.0, 30.0]], "b": [[1.0, 2.0, 3.0]]}
+ ... )
+ >>> result = df.select(
+ ... dfn.functions.array_subtract(
+ ... dfn.col("a"), dfn.col("b")
+ ... ).alias("result")
+ ... )
+ >>> result.collect_column("result")[0].as_py()
+ [9.0, 18.0, 27.0]
+
+
+.. py:function:: array_sum(array: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the sum of the elements of a numeric array.
+
+ NULL elements are skipped. Returns NULL if the array is NULL, empty, or
+ contains only NULL elements.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": [[1.0, None, 3.0]]})
+ >>> result = df.select(
+ ... dfn.functions.array_sum(dfn.col("a")).alias("result")
+ ... )
+ >>> result.collect_column("result")[0].as_py()
+ 4.0
+
+
+.. py:function:: array_to_string(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, null_string: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
Converts each element to its text representation.
+ NULL elements are omitted unless ``null_string`` is given, in which case it
+ is written in their place.
+
.. rubric:: Examples
>>> ctx = dfn.SessionContext()
- >>> df = ctx.from_pydict({"a": [[1, 2, 3]]})
+ >>> df = ctx.from_pydict({"a": [[1, None, 3]]})
>>> result = df.select(
... dfn.functions.array_to_string(dfn.col("a"), ",").alias("s"))
>>> result.collect_column("s")[0].as_py()
- '1,2,3'
+ '1,3'
+
+ >>> result = df.select(
+ ... dfn.functions.array_to_string(
+ ... dfn.col("a"), ",", null_string="*"
+ ... ).alias("s"))
+ >>> result.collect_column("s")[0].as_py()
+ '1,*,3'
.. py:function:: array_transform(array: datafusion.expr.Expr, transform: datafusion.expr.Expr | collections.abc.Callable[Ellipsis, Any]) -> datafusion.expr.Expr
@@ -1632,7 +1844,7 @@ Package Contents
This aggregate function expects a numeric expression and will return a float.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by`` and ``null_treatment``.
:param expression: Values to combine into an array
@@ -1667,16 +1879,17 @@ Package Contents
2.0
-.. py:function:: bit_and(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+.. py:function:: bit_and(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the bitwise AND of the argument.
This aggregate function will bitwise compare every value in the input partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``order_by``, ``null_treatment``, and ``distinct``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the options ``order_by`` and ``null_treatment``.
:param expression: Argument to perform bitwise calculation on
+ :param distinct: If True, evaluate each unique value of expression only once
:param filter: If provided, only compute against rows for which the filter is True
.. rubric:: Examples
@@ -1713,16 +1926,17 @@ Package Contents
8
-.. py:function:: bit_or(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+.. py:function:: bit_or(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the bitwise OR of the argument.
This aggregate function will bitwise compare every value in the input partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``order_by``, ``null_treatment``, and ``distinct``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the options ``order_by`` and ``null_treatment``.
:param expression: Argument to perform bitwise calculation on
+ :param distinct: If True, evaluate each unique value of expression only once
:param filter: If provided, only compute against rows for which the filter is True
.. rubric:: Examples
@@ -1754,7 +1968,7 @@ Package Contents
This aggregate function will bitwise compare every value in the input partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by`` and ``null_treatment``.
:param expression: Argument to perform bitwise calculation on
@@ -1791,7 +2005,7 @@ Package Contents
This aggregate function will compare every value in the input partition. These are
expected to be boolean values.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: Argument to perform calculation on
@@ -1828,7 +2042,7 @@ Package Contents
This aggregate function will compare every value in the input partition. These are
expected to be boolean values.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: Argument to perform calculation on
@@ -1858,9 +2072,9 @@ Package Contents
False
-.. py:function:: btrim(arg: datafusion.expr.Expr) -> datafusion.expr.Expr
+.. py:function:: btrim(arg: datafusion.expr.Expr, characters: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
- Removes all characters, spaces by default, from both sides of a string.
+ Removes ``characters``, spaces by default, from both sides of a string.
.. rubric:: Examples
@@ -1870,6 +2084,15 @@ Package Contents
>>> trim_df.collect_column("trimmed")[0].as_py()
'a'
+ Trim a different set of characters:
+
+ >>> df = ctx.from_pydict({"a": ["xxaxx"]})
+ >>> trim_df = df.select(
+ ... dfn.functions.btrim(dfn.col("a"), characters="x").alias("trimmed")
+ ... )
+ >>> trim_df.collect_column("trimmed")[0].as_py()
+ 'a'
+
.. py:function:: cardinality(array: datafusion.expr.Expr) -> datafusion.expr.Expr
@@ -2076,7 +2299,7 @@ Package Contents
This aggregate function expects both values to be numeric and will return a float.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param value_y: The dependent variable for correlation
@@ -2205,7 +2428,7 @@ Package Contents
This aggregate function will count the non-null rows provided in the expression.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by`` and ``null_treatment``.
:param expressions: Argument to perform bitwise calculation on
@@ -2239,8 +2462,9 @@ Package Contents
This aggregate function will count all of the rows in the partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``order_by``, ``distinct``, and ``null_treatment``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the options ``order_by`` and ``null_treatment``. ``distinct`` counts the distinct
+ values of the constant ``1``, so the result is 1 for any non-empty input.
:param filter: If provided, only count rows for which the filter is True
@@ -2275,7 +2499,7 @@ Package Contents
This aggregate function expects both values to be numeric and will return a float.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param value_y: The dependent variable for covariance
@@ -2314,7 +2538,7 @@ Package Contents
This aggregate function expects both values to be numeric and will return a float.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param value_y: The dependent variable for covariance
@@ -2694,6 +2918,30 @@ Package Contents
6
+.. py:function:: file_row_index() -> datafusion.expr.Expr
+
+ Returns the zero-based position of the current row within its source file.
+
+ The index restarts at zero for each file, so rows from different files in one
+ scan can share a value. Only valid inside a scan of a Parquet table;
+ evaluating it anywhere else raises an error.
+
+ .. rubric:: Examples
+
+ >>> import tempfile, os
+ >>> import pyarrow as pa, pyarrow.parquet as pq
+ >>> tmp = tempfile.mkdtemp()
+ >>> path = os.path.join(tmp, "data.parquet")
+ >>> pq.write_table(pa.table({"a": [10, 20, 30]}), path)
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.read_parquet(path).filter(dfn.col("a") > dfn.lit(10))
+ >>> result = df.select(dfn.functions.file_row_index().alias("i"))
+ >>> result.collect_column("i").to_pylist()
+ [1, 2]
+
+ .. seealso:: :py:func:`input_file_name`.
+
+
.. py:function:: find_in_set(string: datafusion.expr.Expr, string_list: datafusion.expr.Expr | str) -> datafusion.expr.Expr
Find a string in a list of strings.
@@ -2719,7 +2967,7 @@ Package Contents
This aggregate function will return the first value in the partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the option ``distinct``.
:param expression: Argument to perform bitwise calculation on
@@ -2811,38 +3059,40 @@ Package Contents
4
-.. py:function:: gen_series(start: datafusion.expr.Expr, stop: datafusion.expr.Expr, step: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+.. py:function:: gen_series(stop: datafusion.expr.Expr | int) -> datafusion.expr.Expr
+ gen_series(start: datafusion.expr.Expr | int, stop: datafusion.expr.Expr | int, step: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
- Creates a list of values in the range between start and stop.
+ Creates a list of values from ``start`` up to and including ``stop``.
- Unlike :py:func:`range`, this includes the upper bound.
+ Unlike :py:func:`range`, this includes the upper bound. With a single
+ argument it is ``stop`` and the series starts at 0.
.. rubric:: Examples
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [0]})
- >>> result = df.select(
- ... dfn.functions.gen_series(
- ... dfn.lit(1), dfn.lit(5),
- ... ).alias("result"))
+ >>> result = df.select(dfn.functions.gen_series(3).alias("result"))
+ >>> result.collect_column("result")[0].as_py()
+ [0, 1, 2, 3]
+
+ Specify a ``stop``:
+
+ >>> result = df.select(dfn.functions.gen_series(1, stop=5).alias("result"))
>>> result.collect_column("result")[0].as_py()
[1, 2, 3, 4, 5]
- Specify a custom ``step``:
+ Specify a ``step``:
>>> result = df.select(
- ... dfn.functions.gen_series(
- ... dfn.lit(1), dfn.lit(10), step=dfn.lit(3),
- ... ).alias("result"))
+ ... dfn.functions.gen_series(1, stop=10, step=3).alias("result"))
>>> result.collect_column("result")[0].as_py()
[1, 4, 7, 10]
-.. py:function:: generate_series(start: datafusion.expr.Expr, stop: datafusion.expr.Expr, step: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
-
- Creates a list of values in the range between start and stop.
+.. py:function:: generate_series(stop: datafusion.expr.Expr | int) -> datafusion.expr.Expr
+ generate_series(start: datafusion.expr.Expr | int, stop: datafusion.expr.Expr | int, step: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
- Unlike :py:func:`range`, this includes the upper bound.
+ Creates a list of values from ``start`` up to and including ``stop``.
.. seealso:: This is an alias for :py:func:`gen_series`.
@@ -3064,6 +3314,29 @@ Package Contents
True
+.. py:function:: input_file_name() -> datafusion.expr.Expr
+
+ Returns the path of the file that produced the current row.
+
+ Only valid inside a scan of a file-backed table; evaluating it anywhere
+ else raises an error.
+
+ .. rubric:: Examples
+
+ >>> import tempfile, os
+ >>> import pyarrow as pa, pyarrow.parquet as pq
+ >>> tmp = tempfile.mkdtemp()
+ >>> path = os.path.join(tmp, "data.parquet")
+ >>> pq.write_table(pa.table({"a": [1, 2]}), path)
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.read_parquet(path)
+ >>> result = df.select(dfn.functions.input_file_name().alias("f"))
+ >>> result.collect_column("f")[0].as_py().endswith("data.parquet")
+ True
+
+ .. seealso:: :py:func:`file_row_index`.
+
+
.. py:function:: instr(string: datafusion.expr.Expr, substring: datafusion.expr.Expr | str) -> datafusion.expr.Expr
Finds the position from where the ``substring`` matches the ``string``.
@@ -3102,7 +3375,7 @@ Package Contents
True
-.. py:function:: lag(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
+.. py:function:: lag(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment | None = None) -> datafusion.expr.Expr
Create a lag window function.
@@ -3129,6 +3402,8 @@ Package Contents
:param partition_by: Expressions to partition the window frame on.
:param order_by: Set ordering within the window frame. Accepts
column names or expressions.
+ :param null_treatment: Set to ``IGNORE_NULLS`` to skip null values when
+ counting ``shift_offset`` rows.
.. rubric:: Examples
@@ -3153,6 +3428,16 @@ Package Contents
>>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("lag").to_pylist()
[0, 1, 0]
+ >>> df = ctx.from_pydict({"i": [1, 2, 3, 4], "v": [1, None, None, 4]})
+ >>> result = df.select(
+ ... dfn.col("i"),
+ ... dfn.functions.lag(
+ ... dfn.col("v"), order_by="i",
+ ... null_treatment=dfn.common.NullTreatment.IGNORE_NULLS,
+ ... ).alias("lag"))
+ >>> result.sort(dfn.col("i")).collect_column("lag").to_pylist()
+ [None, 1, 1, 1]
+
.. py:function:: lambda_(params: list[str], body: datafusion.expr.Expr) -> datafusion.expr.Expr
@@ -3208,7 +3493,7 @@ Package Contents
This aggregate function will return the last value in the partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the option ``distinct``.
:param expression: Argument to perform bitwise calculation on
@@ -3257,7 +3542,7 @@ Package Contents
12
-.. py:function:: lead(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
+.. py:function:: lead(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment | None = None) -> datafusion.expr.Expr
Create a lead window function.
@@ -3279,7 +3564,7 @@ Package Contents
+--------+------+-----+
To set window function parameters use the window builder approach described in the
- ref:`_window_functions` online documentation.
+ :ref:`window_functions` online documentation.
:param arg: Value to return
:param shift_offset: Number of rows following the current row.
@@ -3287,6 +3572,8 @@ Package Contents
:param partition_by: Expressions to partition the window frame on.
:param order_by: Set ordering within the window frame. Accepts
column names or expressions.
+ :param null_treatment: Set to ``IGNORE_NULLS`` to skip null values when
+ counting ``shift_offset`` rows.
.. rubric:: Examples
@@ -3311,6 +3598,16 @@ Package Contents
>>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("lead").to_pylist()
[2, 0, 0]
+ >>> df = ctx.from_pydict({"i": [1, 2, 3, 4], "v": [1, None, None, 4]})
+ >>> result = df.select(
+ ... dfn.col("i"),
+ ... dfn.functions.lead(
+ ... dfn.col("v"), order_by="i",
+ ... null_treatment=dfn.common.NullTreatment.IGNORE_NULLS,
+ ... ).alias("lead"))
+ >>> result.sort(dfn.col("i")).collect_column("lead").to_pylist()
+ [4, 4, 4, None]
+
.. py:function:: least(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
@@ -3371,6 +3668,13 @@ Package Contents
3
+.. py:function:: list_add(array1: datafusion.expr.Expr, array2: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the element-wise sum of two numeric lists of equal length.
+
+ .. seealso:: This is an alias for :py:func:`array_add`.
+
+
.. py:function:: list_any_match(array: datafusion.expr.Expr, predicate: datafusion.expr.Expr | collections.abc.Callable[Ellipsis, Any]) -> datafusion.expr.Expr
Return ``True`` if any element of a list satisfies a predicate.
@@ -3392,6 +3696,13 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_append`.
+.. py:function:: list_avg(array: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the arithmetic mean of the elements of a numeric list.
+
+ .. seealso:: This is an alias for :py:func:`array_avg`.
+
+
.. py:function:: list_cat(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
Concatenates the input arrays.
@@ -3476,6 +3787,13 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_filter`.
+.. py:function:: list_first(array: datafusion.expr.Expr, predicate: datafusion.expr.Expr | collections.abc.Callable[Ellipsis, Any]) -> datafusion.expr.Expr
+
+ Return the first element of a list for which a predicate is ``True``.
+
+ .. seealso:: This is an alias for :py:func:`array_first`.
+
+
.. py:function:: list_has(array: datafusion.expr.Expr, element: datafusion.expr.Expr) -> datafusion.expr.Expr
Returns true if the element appears in the array, otherwise false.
@@ -3511,7 +3829,7 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_intersect`.
-.. py:function:: list_join(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str) -> datafusion.expr.Expr
+.. py:function:: list_join(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, null_string: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
Converts each element to its text representation.
@@ -3595,6 +3913,13 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_prepend`.
+.. py:function:: list_product(array: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the product of the elements of a numeric list.
+
+ .. seealso:: This is an alias for :py:func:`array_product`.
+
+
.. py:function:: list_push_back(array: datafusion.expr.Expr, element: datafusion.expr.Expr) -> datafusion.expr.Expr
Appends an element to the end of an array.
@@ -3678,6 +4003,13 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_reverse`.
+.. py:function:: list_scale(array: datafusion.expr.Expr, scalar: datafusion.expr.Expr | float) -> datafusion.expr.Expr
+
+ Multiplies each element of a numeric list by a scalar.
+
+ .. seealso:: This is an alias for :py:func:`array_scale`.
+
+
.. py:function:: list_slice(array: datafusion.expr.Expr, begin: datafusion.expr.Expr | int, end: datafusion.expr.Expr | int, stride: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
Returns a slice of the array.
@@ -3692,7 +4024,21 @@ Package Contents
.. seealso:: This is an alias for :py:func:`array_sort`.
-.. py:function:: list_to_string(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str) -> datafusion.expr.Expr
+.. py:function:: list_subtract(array1: datafusion.expr.Expr, array2: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the element-wise difference of two numeric lists of equal length.
+
+ .. seealso:: This is an alias for :py:func:`array_subtract`.
+
+
+.. py:function:: list_sum(array: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Returns the sum of the elements of a numeric list.
+
+ .. seealso:: This is an alias for :py:func:`array_sum`.
+
+
+.. py:function:: list_to_string(expr: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, null_string: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
Converts each element to its text representation.
@@ -3816,9 +4162,9 @@ Package Contents
'...the cat'
-.. py:function:: ltrim(arg: datafusion.expr.Expr) -> datafusion.expr.Expr
+.. py:function:: ltrim(arg: datafusion.expr.Expr, characters: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
- Removes all characters, spaces by default, from the beginning of a string.
+ Removes ``characters``, spaces by default, from the beginning of a string.
.. rubric:: Examples
@@ -3828,6 +4174,15 @@ Package Contents
>>> trim_df.collect_column("trimmed")[0].as_py()
'a '
+ Trim a different set of characters:
+
+ >>> df = ctx.from_pydict({"a": ["xxaxx"]})
+ >>> trim_df = df.select(
+ ... dfn.functions.ltrim(dfn.col("a"), characters="x").alias("trimmed")
+ ... )
+ >>> trim_df.collect_column("trimmed")[0].as_py()
+ 'axx'
+
.. py:function:: make_array(*args: datafusion.expr.Expr) -> datafusion.expr.Expr
@@ -4016,7 +4371,7 @@ Package Contents
Aggregate function that returns the maximum value of the argument.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: The value to find the maximum of
@@ -4055,7 +4410,7 @@ Package Contents
'5d41402abc4b2a76b9719d911017c592'
-.. py:function:: mean(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+.. py:function:: mean(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the average (mean) value of the argument.
@@ -4069,7 +4424,7 @@ Package Contents
This aggregate function returns the median value of the expression for the given
aggregate function.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by`` and ``null_treatment``.
:param expression: The value to compute the median of
@@ -4101,7 +4456,7 @@ Package Contents
Aggregate function that returns the minimum value of the argument.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: The value to find the minimum of
@@ -4190,7 +4545,7 @@ Package Contents
This aggregate function will return the n-th value in the partition.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the option ``distinct``.
:param expression: Argument to perform bitwise calculation on
@@ -4422,18 +4777,21 @@ Package Contents
[0.0, 0.5, 1.0, 0.0, 1.0]
-.. py:function:: percentile_cont(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, percentile: float, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+.. py:function:: percentile_cont(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, percentile: float, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the exact percentile of input values using continuous interpolation.
Unlike :py:func:`approx_percentile_cont`, this function computes the exact
percentile value rather than an approximation.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``order_by``, ``null_treatment``, and ``distinct``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the option ``null_treatment``. A chained ``order_by`` sets only the sort
+ direction; the percentile is still computed over ``sort_expression``, so pass
+ the same expression.
:param sort_expression: Values for which to find the percentile
:param percentile: This must be between 0.0 and 1.0, inclusive
+ :param distinct: If True, duplicate values are removed before computing
:param filter: If provided, only compute against rows for which the filter is True
.. rubric:: Examples
@@ -4455,6 +4813,14 @@ Package Contents
>>> result.collect_column("v")[0].as_py()
3.5
+ >>> df = ctx.from_pydict({"a": [1.0, 1.0, 1.0, 4.0]})
+ >>> result = df.aggregate(
+ ... [], [dfn.functions.percentile_cont(
+ ... dfn.col("a"), 0.5, distinct=True,
+ ... ).alias("v")])
+ >>> result.collect_column("v")[0].as_py()
+ 2.5
+
.. py:function:: pi() -> datafusion.expr.Expr
@@ -4501,7 +4867,7 @@ Package Contents
8.0
-.. py:function:: quantile_cont(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, percentile: float, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
+.. py:function:: quantile_cont(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, percentile: float, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the exact percentile of input values using continuous interpolation.
@@ -4524,6 +4890,13 @@ Package Contents
True
+.. py:function:: rand() -> datafusion.expr.Expr
+
+ Returns a random value in the range ``0.0 <= x < 1.0``.
+
+ .. seealso:: This is an alias for :py:func:`random`.
+
+
.. py:function:: random() -> datafusion.expr.Expr
Returns a random value in the range ``0.0 <= x < 1.0``.
@@ -4540,16 +4913,31 @@ Package Contents
True
-.. py:function:: range(start: datafusion.expr.Expr, stop: datafusion.expr.Expr, step: datafusion.expr.Expr) -> datafusion.expr.Expr
+.. py:function:: range(stop: datafusion.expr.Expr | int) -> datafusion.expr.Expr
+ range(start: datafusion.expr.Expr | int, stop: datafusion.expr.Expr | int, step: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
+
+ Create a list of values from ``start`` up to, but excluding, ``stop``.
- Create a list of values in the range between start and stop.
+ With a single argument it is ``stop`` and the range starts at 0, like
+ Python's built-in :py:class:`range`.
.. rubric:: Examples
>>> ctx = dfn.SessionContext()
>>> df = ctx.from_pydict({"a": [1]})
- >>> result = df.select(
- ... dfn.functions.range(dfn.lit(0), dfn.lit(5), dfn.lit(2)).alias("r"))
+ >>> result = df.select(dfn.functions.range(5).alias("r"))
+ >>> result.collect_column("r")[0].as_py()
+ [0, 1, 2, 3, 4]
+
+ Specify a ``stop``:
+
+ >>> result = df.select(dfn.functions.range(1, stop=5).alias("r"))
+ >>> result.collect_column("r")[0].as_py()
+ [1, 2, 3, 4]
+
+ Specify a ``step``:
+
+ >>> result = df.select(dfn.functions.range(0, stop=5, step=2).alias("r"))
>>> result.collect_column("r")[0].as_py()
[0, 2, 4]
@@ -4773,7 +5161,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -4807,7 +5195,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -4841,7 +5229,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -4875,7 +5263,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -4911,7 +5299,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -4945,7 +5333,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -4979,7 +5367,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -5013,7 +5401,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -5047,7 +5435,7 @@ Package Contents
This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param y: The linear regression dependent variable
@@ -5213,9 +5601,9 @@ Package Contents
'hi!!!'
-.. py:function:: rtrim(arg: datafusion.expr.Expr) -> datafusion.expr.Expr
+.. py:function:: rtrim(arg: datafusion.expr.Expr, characters: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
- Removes all characters, spaces by default, from the end of a string.
+ Removes ``characters``, spaces by default, from the end of a string.
.. rubric:: Examples
@@ -5225,6 +5613,15 @@ Package Contents
>>> trim_df.collect_column("trimmed")[0].as_py()
' a'
+ Trim a different set of characters:
+
+ >>> df = ctx.from_pydict({"a": ["xxaxx"]})
+ >>> trim_df = df.select(
+ ... dfn.functions.rtrim(dfn.col("a"), characters="x").alias("trimmed")
+ ... )
+ >>> trim_df.collect_column("trimmed")[0].as_py()
+ 'xxa'
+
.. py:function:: sha224(arg: datafusion.expr.Expr) -> datafusion.expr.Expr
@@ -5373,7 +5770,7 @@ Package Contents
Computes the standard deviation of the argument.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: The value to find the minimum of
@@ -5403,7 +5800,7 @@ Package Contents
Computes the population standard deviation of the argument.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: The value to find the minimum of
@@ -5439,7 +5836,7 @@ Package Contents
.. seealso:: This is an alias for :py:func:`stddev`.
-.. py:function:: string_agg(expression: datafusion.expr.Expr, delimiter: str, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
+.. py:function:: string_agg(expression: datafusion.expr.Expr, delimiter: str, distinct: bool = False, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Concatenates the input strings.
@@ -5447,11 +5844,12 @@ Package Contents
separating them with the specified delimiter. Non-string values will be converted to
their string equivalents.
- If using the builder functions described in ref:`_aggregation` this function ignores
- the options ``distinct`` and ``null_treatment``.
+ If using the builder functions described in :ref:`aggregation` this function ignores
+ the option ``null_treatment``.
:param expression: Argument to perform bitwise calculation on
:param delimiter: Text to place between each value of expression
+ :param distinct: If True, each unique value of expression is included only once
:param filter: If provided, only compute against rows for which the filter is True
:param order_by: Set the ordering of the expression to evaluate. Accepts
column names or expressions.
@@ -5476,6 +5874,14 @@ Package Contents
>>> result.collect_column("s")[0].as_py()
'y,z'
+ >>> df = ctx.from_pydict({"a": ["y", "x", "y"]})
+ >>> result = df.aggregate(
+ ... [], [dfn.functions.string_agg(
+ ... dfn.col("a"), ",", distinct=True, order_by="a",
+ ... ).alias("s")])
+ >>> result.collect_column("s")[0].as_py()
+ 'x,y'
+
.. py:function:: string_to_array(string: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, null_string: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
@@ -5544,9 +5950,9 @@ Package Contents
True
-.. py:function:: substr(string: datafusion.expr.Expr, position: datafusion.expr.Expr | int) -> datafusion.expr.Expr
+.. py:function:: substr(string: datafusion.expr.Expr, position: datafusion.expr.Expr | int, length: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
- Substring from the ``position`` to the end.
+ Substring from the ``position``, to the end or for ``length`` characters.
.. rubric:: Examples
@@ -5557,6 +5963,13 @@ Package Contents
>>> result.collect_column("s")[0].as_py()
'llo'
+ >>> result = df.select(
+ ... dfn.functions.substr(dfn.col("a"), 2, length=3).alias("s"))
+ >>> result.collect_column("s")[0].as_py()
+ 'ell'
+
+ .. seealso:: :py:func:`substring`.
+
.. py:function:: substr_index(string: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, count: datafusion.expr.Expr | int) -> datafusion.expr.Expr
@@ -5589,13 +6002,20 @@ Package Contents
'hello'
+.. py:function:: substring_index(string: datafusion.expr.Expr, delimiter: datafusion.expr.Expr | str, count: datafusion.expr.Expr | int) -> datafusion.expr.Expr
+
+ Returns an indexed substring.
+
+ .. seealso:: This is an alias for :py:func:`substr_index`.
+
+
.. py:function:: sum(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sum of a set of numbers.
This aggregate function expects a numeric expression.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by`` and ``null_treatment``.
:param expression: Values to combine into an array
@@ -5948,9 +6368,9 @@ Package Contents
'HELLO'
-.. py:function:: trim(arg: datafusion.expr.Expr) -> datafusion.expr.Expr
+.. py:function:: trim(arg: datafusion.expr.Expr, characters: datafusion.expr.Expr | str | None = None) -> datafusion.expr.Expr
- Removes all characters, spaces by default, from both sides of a string.
+ Removes ``characters``, spaces by default, from both sides of a string.
.. rubric:: Examples
@@ -5960,6 +6380,15 @@ Package Contents
>>> result.collect_column("t")[0].as_py()
'hello'
+ Trim a different set of characters:
+
+ >>> df = ctx.from_pydict({"a": ["xxhelloxx"]})
+ >>> result = df.select(
+ ... dfn.functions.trim(dfn.col("a"), characters="x").alias("t")
+ ... )
+ >>> result.collect_column("t")[0].as_py()
+ 'hello'
+
.. py:function:: trunc(num: datafusion.expr.Expr, precision: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
@@ -6090,7 +6519,7 @@ Package Contents
Computes the population variance of the argument.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: The variable to compute the variance for
@@ -6127,7 +6556,7 @@ Package Contents
Computes the sample variance of the argument.
- If using the builder functions described in ref:`_aggregation` this function ignores
+ If using the builder functions described in :ref:`aggregation` this function ignores
the options ``order_by``, ``null_treatment``, and ``distinct``.
:param expression: The variable to compute the variance for
diff --git a/_sources/autoapi/datafusion/functions/spark/index.rst.txt b/_sources/autoapi/datafusion/functions/spark/index.rst.txt
index a2eae89b..cccbbd5a 100644
--- a/_sources/autoapi/datafusion/functions/spark/index.rst.txt
+++ b/_sources/autoapi/datafusion/functions/spark/index.rst.txt
@@ -30,6 +30,7 @@ Functions
datafusion.functions.spark.array_contains
datafusion.functions.spark.array_repeat
datafusion.functions.spark.ascii
+ datafusion.functions.spark.atan2
datafusion.functions.spark.avg
datafusion.functions.spark.base64
datafusion.functions.spark.bin
@@ -40,10 +41,14 @@ Functions
datafusion.functions.spark.bitmap_count
datafusion.functions.spark.bitwise_not
datafusion.functions.spark.ceil
+ datafusion.functions.spark.ceiling
datafusion.functions.spark.char
+ datafusion.functions.spark.char_length
+ datafusion.functions.spark.character_length
datafusion.functions.spark.collect_list
datafusion.functions.spark.collect_set
datafusion.functions.spark.concat
+ datafusion.functions.spark.concat_ws
datafusion.functions.spark.crc32
datafusion.functions.spark.csc
datafusion.functions.spark.date_add
@@ -51,14 +56,19 @@ Functions
datafusion.functions.spark.date_part
datafusion.functions.spark.date_sub
datafusion.functions.spark.date_trunc
+ datafusion.functions.spark.dateadd
+ datafusion.functions.spark.datediff
+ datafusion.functions.spark.datepart
datafusion.functions.spark.elt
datafusion.functions.spark.expm1
datafusion.functions.spark.factorial
datafusion.functions.spark.floor
datafusion.functions.spark.format_string
datafusion.functions.spark.from_utc_timestamp
+ datafusion.functions.spark.getbit
datafusion.functions.spark.hex
datafusion.functions.spark.hour
+ datafusion.functions.spark.hypot
datafusion.functions.spark.if_
datafusion.functions.spark.ilike
datafusion.functions.spark.is_valid_utf8
@@ -74,14 +84,20 @@ Functions
datafusion.functions.spark.map_from_entries
datafusion.functions.spark.minute
datafusion.functions.spark.modulus
+ datafusion.functions.spark.monthname
datafusion.functions.spark.negative
datafusion.functions.spark.next_day
datafusion.functions.spark.parse_url
datafusion.functions.spark.pmod
+ datafusion.functions.spark.pow
+ datafusion.functions.spark.power
+ datafusion.functions.spark.printf
+ datafusion.functions.spark.quote
datafusion.functions.spark.rint
datafusion.functions.spark.round
datafusion.functions.spark.sec
datafusion.functions.spark.second
+ datafusion.functions.spark.sha
datafusion.functions.spark.sha1
datafusion.functions.spark.sha2
datafusion.functions.spark.shiftleft
@@ -94,6 +110,7 @@ Functions
datafusion.functions.spark.space
datafusion.functions.spark.spark_cast
datafusion.functions.spark.str_to_map
+ datafusion.functions.spark.substr
datafusion.functions.spark.substring
datafusion.functions.spark.time_trunc
datafusion.functions.spark.to_utc_timestamp
@@ -109,6 +126,7 @@ Functions
datafusion.functions.spark.unix_seconds
datafusion.functions.spark.url_decode
datafusion.functions.spark.url_encode
+ datafusion.functions.spark.weekday
datafusion.functions.spark.width_bucket
datafusion.functions.spark.xxhash64
@@ -212,6 +230,22 @@ Module Contents
65
+.. py:function:: atan2(col1: datafusion.expr.Expr | float, col2: datafusion.expr.Expr | float) -> datafusion.expr.Expr
+
+ Spark ``atan2``: angle in radians of the point ``(col2, col1)``.
+
+ ``col1`` is the y coordinate and ``col2`` the x coordinate. Both accept
+ native numbers or :class:`Expr`.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> r = df.select(dfn.functions.spark.atan2(1.0, 0.0).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 1.5707963267948966
+
+
.. py:function:: avg(col: datafusion.expr.Expr, distinct: bool | None = None, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment | None = None) -> datafusion.expr.Expr
Spark ``avg``: returns the mean of a numeric column.
@@ -350,6 +384,13 @@ Module Contents
2
+.. py:function:: ceiling(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``ceiling``: smallest integer ≥ arg.
+
+ .. seealso:: This is an alias for :py:func:`ceil`.
+
+
.. py:function:: char(col: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``char``: ASCII character for a code point (mod 256).
@@ -363,6 +404,20 @@ Module Contents
'A'
+.. py:function:: char_length(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``char_length``: character length of a string, or bytes of binary.
+
+ .. seealso:: This is an alias for :py:func:`length`.
+
+
+.. py:function:: character_length(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``character_length``: character length of a string, or bytes of binary.
+
+ .. seealso:: This is an alias for :py:func:`length`.
+
+
.. py:function:: collect_list(col: datafusion.expr.Expr, distinct: bool | None = None, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment | None = None) -> datafusion.expr.Expr
Spark ``collect_list``: collect values into an array (preserves dups).
@@ -405,6 +460,25 @@ Module Contents
'ab'
+.. py:function:: concat_ws(sep: datafusion.expr.Expr | str, *cols: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``concat_ws``: joins strings and arrays of strings with ``sep``.
+
+ NULL inputs are skipped rather than propagated.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": ["x"], "b": [None], "c": ["z"]})
+ >>> r = df.select(
+ ... dfn.functions.spark.concat_ws(
+ ... "-", dfn.col("a"), dfn.col("b"), dfn.col("c")
+ ... ).alias("v")
+ ... )
+ >>> r.collect_column("v")[0].as_py()
+ 'x-z'
+
+
.. py:function:: crc32(col: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``crc32``: cyclic redundancy check value as a bigint.
@@ -524,6 +598,27 @@ Module Contents
datetime.datetime(2020, 1, 1, 0, 0)
+.. py:function:: dateadd(start: datafusion.expr.Expr, days: datafusion.expr.Expr | int) -> datafusion.expr.Expr
+
+ Spark ``dateadd``: date + N days.
+
+ .. seealso:: This is an alias for :py:func:`date_add`.
+
+
+.. py:function:: datediff(end: datafusion.expr.Expr, start: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``datediff``: number of days from ``start`` to ``end``.
+
+ .. seealso:: This is an alias for :py:func:`date_diff`.
+
+
+.. py:function:: datepart(field: datafusion.expr.Expr | str, source: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``datepart``: extract ``field`` from a date/time/timestamp.
+
+ .. seealso:: This is an alias for :py:func:`date_part`.
+
+
.. py:function:: elt(*inputs: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``elt``: returns the n-th input (1-indexed).
@@ -626,6 +721,13 @@ Module Contents
datetime.datetime(2020, 1, 15, 14, 30, 45)
+.. py:function:: getbit(col: datafusion.expr.Expr, pos: datafusion.expr.Expr | str) -> datafusion.expr.Expr
+
+ Spark ``getbit``: returns the bit (0 or 1) at ``pos``.
+
+ .. seealso:: This is an alias for :py:func:`bit_get`.
+
+
.. py:function:: hex(col: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``hex``: hexadecimal representation.
@@ -657,6 +759,21 @@ Module Contents
14
+.. py:function:: hypot(col1: datafusion.expr.Expr | float, col2: datafusion.expr.Expr | float) -> datafusion.expr.Expr
+
+ Spark ``hypot``: ``sqrt(col1^2 + col2^2)`` without intermediate overflow.
+
+ Both arguments accept native numbers or :class:`Expr`.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> r = df.select(dfn.functions.spark.hypot(3.0, 4.0).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 5.0
+
+
.. py:function:: if_(condition: datafusion.expr.Expr, if_true: datafusion.expr.Expr | Any, if_false: datafusion.expr.Expr | Any) -> datafusion.expr.Expr
Spark ``if``: returns ``if_true`` when ``condition`` is true, else ``if_false``.
@@ -729,7 +846,7 @@ Module Contents
{'c0': '1', 'c1': 'x'}
-.. py:function:: last_day(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+.. py:function:: last_day(date: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``last_day``: last day of the month containing the date.
@@ -927,6 +1044,22 @@ Module Contents
1
+.. py:function:: monthname(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``monthname``: three-letter abbreviated month name.
+
+ .. rubric:: Examples
+
+ >>> import pyarrow as pa
+ >>> from datetime import date
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> d = dfn.lit(pa.scalar(date(2024, 3, 15)))
+ >>> r = df.select(dfn.functions.spark.monthname(d).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 'Mar'
+
+
.. py:function:: negative(col: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``negative``: unary minus.
@@ -1007,6 +1140,63 @@ Module Contents
2
+.. py:function:: pow(col1: datafusion.expr.Expr | float, col2: datafusion.expr.Expr | float) -> datafusion.expr.Expr
+
+ Spark ``pow``: ``col1`` raised to the power ``col2``, as a double.
+
+ Both arguments accept native numbers or :class:`Expr`.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> r = df.select(dfn.functions.spark.pow(2, 10).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 1024.0
+
+
+.. py:function:: power(col1: datafusion.expr.Expr | float, col2: datafusion.expr.Expr | float) -> datafusion.expr.Expr
+
+ Spark ``power``: ``col1`` raised to the power ``col2``.
+
+ .. seealso:: This is an alias for :py:func:`pow`.
+
+
+.. py:function:: printf(format: datafusion.expr.Expr | str, *cols: datafusion.expr.Expr | str) -> datafusion.expr.Expr
+
+ Spark ``printf``: printf-style format string.
+
+ Unlike :py:func:`format_string`, a bare ``str`` ``format`` or ``cols`` entry
+ is treated as a column name (matching pyspark), not a literal; pass
+ :func:`~datafusion.lit` for a literal format.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"a": ["aa%d%s"], "b": [123], "c": ["cc"]})
+ >>> r = df.select(dfn.functions.spark.printf("a", "b", "c").alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 'aa123cc'
+
+ >>> r = df.select(
+ ... dfn.functions.spark.printf(dfn.lit("%d-%s"), "b", "c").alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ '123-cc'
+
+
+.. py:function:: quote(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``quote``: wraps a string in single quotes, escaping inner quotes.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> r = df.select(dfn.functions.spark.quote(dfn.lit("it's")).alias("v"))
+ >>> print(r.collect_column("v")[0].as_py())
+ 'it\'s'
+
+
.. py:function:: rint(col: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``rint``: round to nearest mathematical integer (as double).
@@ -1072,6 +1262,13 @@ Module Contents
45
+.. py:function:: sha(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``sha``: SHA-1 hash as a hex string.
+
+ .. seealso:: This is an alias for :py:func:`sha1`.
+
+
.. py:function:: sha1(col: datafusion.expr.Expr) -> datafusion.expr.Expr
Spark ``sha1``: SHA-1 hash as a hex string.
@@ -1285,6 +1482,27 @@ Module Contents
[('a', '1'), ('b', '2')]
+.. py:function:: substr(str: datafusion.expr.Expr, pos: datafusion.expr.Expr | int, len: datafusion.expr.Expr | int | None = None) -> datafusion.expr.Expr
+
+ Spark ``substr``: 1-indexed substring, to the end when ``len`` is omitted.
+
+ Same as :py:func:`substring` except that ``len`` is optional. ``pos`` and
+ ``len`` accept native ``int`` values or :class:`Expr`.
+
+ .. rubric:: Examples
+
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> r = df.select(dfn.functions.spark.substr(dfn.lit("hello"), 2).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 'ello'
+
+ >>> r = df.select(
+ ... dfn.functions.spark.substr(dfn.lit("hello"), 2, len=3).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 'ell'
+
+
.. py:function:: substring(str: datafusion.expr.Expr, pos: datafusion.expr.Expr | int, len: datafusion.expr.Expr | int) -> datafusion.expr.Expr
Spark ``substring``: 1-indexed substring starting at ``pos`` of given ``length``.
@@ -1534,6 +1752,22 @@ Module Contents
'a+b'
+.. py:function:: weekday(col: datafusion.expr.Expr) -> datafusion.expr.Expr
+
+ Spark ``weekday``: day of the week, Monday = 0 through Sunday = 6.
+
+ .. rubric:: Examples
+
+ >>> import pyarrow as pa
+ >>> from datetime import date
+ >>> ctx = dfn.SessionContext()
+ >>> df = ctx.from_pydict({"x": [1]})
+ >>> d = dfn.lit(pa.scalar(date(2024, 3, 15)))
+ >>> r = df.select(dfn.functions.spark.weekday(d).alias("v"))
+ >>> r.collect_column("v")[0].as_py()
+ 4
+
+
.. py:function:: width_bucket(v: datafusion.expr.Expr, min: datafusion.expr.Expr, max: datafusion.expr.Expr, numBucket: datafusion.expr.Expr | int) -> datafusion.expr.Expr
Spark ``width_bucket``: bucket number for ``value`` in equi-width histogram.
diff --git a/_sources/autoapi/datafusion/index.rst.txt b/_sources/autoapi/datafusion/index.rst.txt
index f3039a74..45d0c2c2 100644
--- a/_sources/autoapi/datafusion/index.rst.txt
+++ b/_sources/autoapi/datafusion/index.rst.txt
@@ -1061,8 +1061,9 @@ Package Contents
Only evaluate distinct values for an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on a window
+ function unless that is an aggregate used as one. If used on any other type of
+ expression, an error will be generated when ``build()`` is called.
@@ -1101,8 +1102,9 @@ Package Contents
Filter an aggregate function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on a window
+ function unless that is an aggregate used as one. If used on any other type of
+ expression, an error will be generated when ``build()`` is called.
@@ -1334,12 +1336,19 @@ Package Contents
.. py:method:: over(window: Window) -> Expr
- Turn an aggregate function into a window function.
+ Evaluate an aggregate or window function over a window.
- This function turns any aggregate function into a window function. With the
+ On an aggregate function this turns it into a window function. With the
exception of ``partition_by``, how each of the parameters is used is determined
by the underlying aggregate function.
+ On an aggregate, the ``null_treatment``, ``filter``, and ``distinct`` options
+ it was built with are kept, and an ``order_by`` raises; see
+ :ref:`aggregate_over_options`.
+
+ On a window function that already has a ``partition_by``, ``order_by``, or
+ ``window_frame``, this raises; see :ref:`window_function_chaining`.
+
:param window: Window definition
@@ -1349,8 +1358,10 @@ Package Contents
Set the partitioning for a window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on an
+ aggregate function that has not been turned into a window function with
+ :py:meth:`over`. If used on any other type of expression, an error will be
+ generated when ``build()`` is called.
@@ -1644,11 +1655,13 @@ Package Contents
.. py:method:: window_frame(window_frame: WindowFrame) -> ExprFuncBuilder
- Set the frame fora window function.
+ Set the frame for a window function.
This function will create an :py:class:`ExprFuncBuilder` that can be used to
- set parameters for either window or aggregate functions. If used on any other
- type of expression, an error will be generated when ``build()`` is called.
+ set parameters for either window or aggregate functions. It raises on an
+ aggregate function that has not been turned into a window function with
+ :py:meth:`over`. If used on any other type of expression, an error will be
+ generated when ``build()`` is called.
diff --git a/_sources/autoapi/datafusion/user_defined/index.rst.txt b/_sources/autoapi/datafusion/user_defined/index.rst.txt
index 415240b8..9bc62295 100644
--- a/_sources/autoapi/datafusion/user_defined/index.rst.txt
+++ b/_sources/autoapi/datafusion/user_defined/index.rst.txt
@@ -142,13 +142,13 @@ Module Contents
Wrap an already-constructed internal ``AggregateUDF`` handle.
- Used by :py:meth:`SessionContext.udaf` to surface a function looked
- up from the session's function registry without re-running
- :py:meth:`__init__`.
+ Used by :py:meth:`SessionContext.udaf` and :py:meth:`from_pycapsule`
+ to wrap a handle from the session's function registry or an FFI
+ capsule without re-running :py:meth:`__init__`.
- .. py:method:: from_pycapsule(func: AggregateUDFExportable | _typeshed.CapsuleType) -> AggregateUDF
+ .. py:method:: from_pycapsule(func: AggregateUDFExportable | types.CapsuleType) -> AggregateUDF
:staticmethod:
@@ -162,7 +162,7 @@ Module Contents
.. py:method:: udaf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, AggregateUDF]
udaf(accum: collections.abc.Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) -> AggregateUDF
udaf(accum: AggregateUDFExportable) -> AggregateUDF
- udaf(accum: _typeshed.CapsuleType) -> AggregateUDF
+ udaf(accum: types.CapsuleType) -> AggregateUDF
:staticmethod:
@@ -392,13 +392,13 @@ Module Contents
Wrap an already-constructed internal ``ScalarUDF`` handle.
- Used by :py:meth:`SessionContext.udf` to surface a function looked
- up from the session's function registry without re-running
- :py:meth:`__init__`.
+ Used by :py:meth:`SessionContext.udf` and :py:meth:`from_pycapsule`
+ to wrap a handle from the session's function registry or an FFI
+ capsule without re-running :py:meth:`__init__`.
- .. py:method:: from_pycapsule(func: ScalarUDFExportable) -> ScalarUDF
+ .. py:method:: from_pycapsule(func: ScalarUDFExportable | types.CapsuleType) -> ScalarUDF
:staticmethod:
@@ -412,6 +412,7 @@ Module Contents
.. py:method:: udf(input_fields: collections.abc.Sequence[pyarrow.DataType | pyarrow.Field] | pyarrow.DataType | pyarrow.Field, return_field: pyarrow.DataType | pyarrow.Field, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, ScalarUDF]
udf(func: collections.abc.Callable[Ellipsis, _R], input_fields: collections.abc.Sequence[pyarrow.DataType | pyarrow.Field] | pyarrow.DataType | pyarrow.Field, return_field: pyarrow.DataType | pyarrow.Field, volatility: Volatility | str, name: str | None = None) -> ScalarUDF
udf(func: ScalarUDFExportable) -> ScalarUDF
+ udf(func: types.CapsuleType) -> ScalarUDF
:staticmethod:
@@ -871,9 +872,9 @@ Module Contents
Wrap an already-constructed internal ``WindowUDF`` handle.
- Used by :py:meth:`SessionContext.udwf` to surface a function looked
- up from the session's function registry without re-running
- :py:meth:`__init__`.
+ Used by :py:meth:`SessionContext.udwf` and :py:meth:`from_pycapsule`
+ to wrap a handle from the session's function registry or an FFI
+ capsule without re-running :py:meth:`__init__`.
@@ -893,7 +894,7 @@ Module Contents
- .. py:method:: from_pycapsule(func: WindowUDFExportable) -> WindowUDF
+ .. py:method:: from_pycapsule(func: WindowUDFExportable | types.CapsuleType) -> WindowUDF
:staticmethod:
@@ -906,6 +907,8 @@ Module Contents
.. py:method:: udwf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, WindowUDF]
udwf(func: collections.abc.Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> WindowUDF
+ udwf(func: WindowUDFExportable) -> WindowUDF
+ udwf(func: types.CapsuleType) -> WindowUDF
:staticmethod:
@@ -997,7 +1000,7 @@ Module Contents
.. py:method:: __datafusion_window_udf__() -> object
-.. py:function:: _is_pycapsule(value: object) -> TypeGuard[_typeshed.CapsuleType]
+.. py:function:: _is_pycapsule(value: object) -> TypeGuard[types.CapsuleType]
Return ``True`` when ``value`` is a CPython ``PyCapsule``.
diff --git a/_sources/user-guide/common-operations/aggregations.md.txt b/_sources/user-guide/common-operations/aggregations.md.txt
index 9c2d58e3..0e073185 100644
--- a/_sources/user-guide/common-operations/aggregations.md.txt
+++ b/_sources/user-guide/common-operations/aggregations.md.txt
@@ -393,11 +393,12 @@ The available aggregate functions are:
- {py:func}`datafusion.functions.regr_avgy`
- {py:func}`datafusion.functions.regr_sxx`
- {py:func}`datafusion.functions.regr_syy`
- - {py:func}`datafusion.functions.regr_slope`
+ - {py:func}`datafusion.functions.regr_sxy`
07. Positional Functions
: - {py:func}`datafusion.functions.first_value`
- {py:func}`datafusion.functions.last_value`
- {py:func}`datafusion.functions.nth_value`
+ - {py:func}`datafusion.functions.any_value`
08. String Functions
: - {py:func}`datafusion.functions.string_agg`
09. Percentile Functions
diff --git a/_sources/user-guide/common-operations/expressions.md.txt b/_sources/user-guide/common-operations/expressions.md.txt
index 7985705e..ab42172a 100644
--- a/_sources/user-guide/common-operations/expressions.md.txt
+++ b/_sources/user-guide/common-operations/expressions.md.txt
@@ -157,7 +157,9 @@ In this example, the `repeated_array` column will contain `[[1, 2, 3], [1, 2, 3]
Some array functions take a *lambda function*: a small function that runs once
per element. {py:func}`~datafusion.functions.array_transform` maps a lambda over
every element, {py:func}`~datafusion.functions.array_filter` keeps the elements
-for which a predicate lambda is true, and
+for which a predicate lambda is true,
+{py:func}`~datafusion.functions.array_first` returns the first element that
+satisfies a predicate lambda, and
{py:func}`~datafusion.functions.array_any_match` returns whether any element
satisfies a predicate lambda. (Functions that take another function as an
argument are sometimes called *higher-order* functions.)
@@ -173,6 +175,7 @@ ctx = SessionContext()
df = ctx.from_pydict({"a": [[1, 2, 3], [4, 5]]})
df.select(f.array_transform(col("a"), lambda v: v * 2).alias("doubled"))
df.select(f.array_filter(col("a"), lambda v: v > 2).alias("big_only"))
+df.select(f.array_first(col("a"), lambda v: v > 2).alias("first_big"))
df.select(f.array_any_match(col("a"), lambda v: v > 3).alias("has_big"))
```
diff --git a/_sources/user-guide/common-operations/functions.md.txt b/_sources/user-guide/common-operations/functions.md.txt
index 9939e2f0..d5ddffb7 100644
--- a/_sources/user-guide/common-operations/functions.md.txt
+++ b/_sources/user-guide/common-operations/functions.md.txt
@@ -158,3 +158,38 @@ df = df.fill_null("missing", subset=["name", "category"])
```
The fill value will be cast to match each column's type. If casting fails for a column, that column remains unchanged.
+
+## fill_nan
+
+The `fill_nan()` method replaces NaN values in floating-point columns. NaN is
+distinct from NULL, which `fill_null()` handles:
+
+```python
+# Replace NaN with 0.0 in every floating-point column
+df = df.fill_nan(0.0)
+
+# Replace NaN only in specific columns
+df = df.fill_nan(0.0, subset=["price"])
+```
+
+(fill_column_names)=
+
+## Column names with uppercase letters or dots
+
+`fill_null()` and `fill_nan()` fail on a DataFrame that has any column whose
+name contains an uppercase letter or a `.`, even when that column is not in
+`subset`
+([apache/datafusion#25829](https://github.com/apache/datafusion/issues/25829)):
+
+```python
+df = ctx.from_pydict({"Price": [float("nan"), 2.0], "qty": [float("nan"), 1.0]})
+df.fill_nan(0.0, subset=["qty"])
+# Schema error: No field named price. Did you mean '..."Price"'?
+```
+
+Until that is fixed, rename such columns first. Quote the old name so it is
+not normalized:
+
+```python
+df.with_column_renamed('"Price"', "price").fill_nan(0.0, subset=["qty"])
+```
diff --git a/_sources/user-guide/common-operations/windows.md.txt b/_sources/user-guide/common-operations/windows.md.txt
index bee96c82..31fcdbe0 100644
--- a/_sources/user-guide/common-operations/windows.md.txt
+++ b/_sources/user-guide/common-operations/windows.md.txt
@@ -135,11 +135,54 @@ df.select(
)
```
+(window_function_chaining)=
+
+#### Chaining onto a window function
+
+Set a window function's `partition_by`, `order_by`, and `window_frame` in one
+place: its keyword arguments, a single `over()`, or one builder chain ending in
+`build()`. Chaining a builder method or `over()` onto a window function that
+already has any of them raises:
+
+```python
+# Raises: lead already has window options (order_by)
+f.lead(col("v"), order_by="t").over(Window(partition_by=[col("g")]))
+
+# Set them together instead.
+f.lead(col("v")).over(Window(partition_by=[col("g")], order_by="t"))
+```
+
+A built window function stores a concrete frame with no record of whether you
+chose it or it was derived from `order_by`, so the options cannot be merged
+without guessing. Merging may become possible once
+[apache/datafusion#25934](https://github.com/apache/datafusion/issues/25934)
+is resolved.
+
+The `null_treatment` already set is kept, as are `filter` and `distinct` on an
+aggregate used as a window function. The whole-partition frame counts as no
+frame, so adding an `order_by` derives the running frame, even when you passed
+that frame explicitly:
+
+```python
+whole = WindowFrame("rows", None, None) # same as the no-order_by default
+
+# Both give a running sum.
+f.sum(col("v")).over(Window()).order_by(col("v")).build()
+f.sum(col("v")).over(Window(window_frame=whole)).order_by(col("v")).build()
+```
+
+To keep that frame, set it after the `order_by`, or pass both in one `Window`:
+
+```python
+f.sum(col("v")).over(Window()).order_by(col("v")).window_frame(whole).build()
+f.sum(col("v")).over(Window(order_by=col("v"), window_frame=whole))
+```
+
### Null Treatment
When using aggregate functions as window functions, it is often useful to specify how null values
-should be treated. In order to do this you need to use the builder function. In future releases
-we expect this to be simplified in the interface.
+should be treated. Pass `null_treatment` in the `Window`, or set it on the aggregate itself, which
+`over()` keeps (see {ref}`aggregate_over_options`).
One common usage for handling nulls is the case where you want to find the last value up to the
current row. In the following example we demonstrate how setting the null treatment to ignore
@@ -196,6 +239,30 @@ df.select(
)
```
+(aggregate_over_options)=
+
+### Options set on the aggregate
+
+`over()` keeps the `filter`, `distinct`, and `null_treatment` options an
+aggregate was built with:
+
+```python
+# Averages the distinct values 1.0 and 4.0.
+f.avg(col("v"), distinct=True).over(Window())
+```
+
+An aggregate's `order_by` raises, as `ORDER BY` inside an aggregate call does
+with `OVER` in SQL. A window does not pass an ordering to the aggregate, so
+the `order_by` in the `Window` only sets the frame and the order of rows. The
+one exception is a `WITHIN GROUP` function such as
+{py:func}`~datafusion.functions.percentile_cont`, which computes ascending as a
+window: an ascending `sort_expression` is accepted, and a descending one raises.
+
+```python
+f.percentile_cont(col("v"), 0.25).over(Window()) # ascending, accepted
+f.percentile_cont(col("v").sort(ascending=False), 0.25).over(Window()) # raises
+```
+
## Available Functions
The possible window functions are:
diff --git a/_sources/user-guide/upgrade-guides.md.txt b/_sources/user-guide/upgrade-guides.md.txt
index 0b8e94bb..3194eb61 100644
--- a/_sources/user-guide/upgrade-guides.md.txt
+++ b/_sources/user-guide/upgrade-guides.md.txt
@@ -198,6 +198,179 @@ ctx.execute(plan, partitions=0) # before
ctx.execute(plan, partition=0) # after
```
+### More aggregate functions accept `distinct`
+
+{py:func}`~datafusion.functions.bit_and`,
+{py:func}`~datafusion.functions.bit_or`,
+{py:func}`~datafusion.functions.mean`,
+{py:func}`~datafusion.functions.percentile_cont`,
+{py:func}`~datafusion.functions.quantile_cont`, and
+{py:func}`~datafusion.functions.string_agg` now accept a `distinct` argument.
+As with `sum` and `avg` in 54.0.0, `distinct` is inserted *before* `filter`, so
+code that passed `filter` (or, for `string_agg`, `order_by`) positionally must
+pass it by keyword.
+
+```python
+f.bit_and(column("a"), my_filter) # before
+f.bit_and(column("a"), filter=my_filter) # after
+```
+
+Passing `filter` to `mean` previously raised a `TypeError`, whether passed
+positionally or by keyword; it now works when passed by keyword.
+
+### Chaining keeps options already set
+
+Chaining a builder method (`order_by`, `filter`, `distinct`, `null_treatment`,
+`partition_by`, `window_frame`) or `over()` onto a function used to start from
+an empty builder, so options set by the function's keyword arguments were
+silently reset. On an aggregate they are now kept, which can change results:
+
+```python
+e = f.string_agg(col("s"), ",", order_by="s")
+e.distinct().build() # before: order_by dropped; after: kept
+```
+
+On a window function that already has a `partition_by`, `order_by`, or
+`window_frame`, chaining now raises instead of dropping them. Set them in one
+place, as described in {ref}`window_function_chaining`:
+
+```python
+e = f.lead(col("v"), 1, partition_by=[col("g")], order_by="t")
+e.over(Window(order_by="t")) # before: partition dropped; after: raises
+f.lead(col("v"), 1).over(Window(partition_by=[col("g")], order_by="t")) # after
+```
+
+Options that used to be dropped now take effect, so a chain that ran before
+may now raise. For example, DISTINCT requires the ORDER BY expressions to be
+among the arguments:
+
+```python
+f.array_agg(col("s"), distinct=True).order_by(col("v")).build()
+# before: ran without DISTINCT
+# after: Execution error: In an aggregate with DISTINCT, ORDER BY expressions
+# must appear in argument list
+```
+
+Drop `distinct`, or order by the aggregated column, to get either of the
+results the chain can actually produce.
+
+An option that does not apply to the function now raises as soon as it is
+set, anywhere in the chain. `filter` and `distinct` need an aggregate,
+including one used as a window function, and `partition_by` and
+`window_frame` need a window function. Later in a chain these were silently
+dropped:
+
+```python
+f.sum(col("v")).filter(col("v") > lit(1)).partition_by(col("g"))
+# before: partition_by dropped; after: raises
+```
+
+The default `RESPECT NULLS` set by `first_value`, `last_value`, and `nth_value`
+is also kept, so their generated column names change:
+
+```python
+f.first_value(col("a")).order_by(col("b")).build()
+# before: first_value(a) ORDER BY [b ASC NULLS FIRST]
+# after: first_value(a) RESPECT NULLS ORDER BY [b ASC NULLS FIRST]
+```
+
+This now matches the name from `f.first_value(col("a"), order_by=col("b"))`.
+Code that selects the result by its generated name should `alias()` it instead.
+
+### `over()` keeps options set on an aggregate
+
+`Expr.over()` on an aggregate used to drop the `filter`, `distinct`,
+`null_treatment`, and `order_by` options it was built with. The first three are
+now kept, which can change results:
+
+```python
+f.avg(col("v"), distinct=True).over(Window())
+# v = [1, 1, 4]; before: 2.0, after: 2.5
+```
+
+An `order_by` on the aggregate now raises instead of being dropped, as it does
+with `OVER` in SQL. Remove it, or move it into the `Window` if it was meant to
+order the rows:
+
+```python
+# before: order_by dropped; after: raises
+f.first_value(col("v"), order_by=col("i").sort(ascending=False)).over(
+ Window(partition_by=[col("g")])
+)
+
+# after: the Window orders the rows the aggregate sees
+f.first_value(col("v")).over(
+ Window(partition_by=[col("g")], order_by=[col("i").sort(ascending=False)])
+)
+```
+
+A `WITHIN GROUP` function such as `percentile_cont` still accepts
+an ascending `sort_expression`, and raises on a descending one, which used to
+give the ascending result. See {ref}`aggregate_over_options`.
+
+### Python aggregate UDFs reject `DISTINCT`
+
+A Python {py:class}`~datafusion.user_defined.Accumulator` cannot deduplicate its
+input, so a Python aggregate UDF with `DISTINCT` counted every row. It now
+raises instead of returning that result:
+
+```python
+my_sum(col("v")).over(Window()).distinct().build()
+# v = [1, 1, 1, 5]; before: 8.0; after: DISTINCT is not supported ...
+```
+
+When the optimizer rewrites the query to group by the distinct values first,
+as it does for SQL's `SELECT my_sum(DISTINCT v) FROM t`, the query still runs
+and gives the distinct result.
+
+### Percentile functions keep the sort direction
+
+{py:func}`~datafusion.functions.percentile_cont`,
+{py:func}`~datafusion.functions.quantile_cont`,
+{py:func}`~datafusion.functions.approx_percentile_cont`, and
+{py:func}`~datafusion.functions.approx_percentile_cont_with_weight` ignored the
+direction of `sort_expression`, so a descending sort gave the ascending result.
+Used as aggregates, they now match `WITHIN GROUP (ORDER BY ... DESC)` in SQL
+(see {ref}`aggregate_over_options` for their use in a window):
+
+```python
+f.percentile_cont(col("a").sort(ascending=False), 0.25)
+# a = [1, 2, 3, 4, 5]; before: 2.0, after: 4.0
+```
+
+Their generated column names now include the ordering, in the same form as SQL.
+Code that selects the result by its generated name should `alias()` it instead.
+
+```python
+f.percentile_cont(col("a"), 0.25)
+# before: percentile_cont(t.a,Float64(0.25))
+# after: percentile_cont(Float64(0.25)) WITHIN GROUP [t.a ASC NULLS FIRST]
+```
+
+### `fill_null(subset=[])` fills no columns
+
+{py:meth}`~datafusion.dataframe.DataFrame.fill_null` with an empty `subset`
+list used to fill every column, the same as `subset=None`. It now returns the
+DataFrame unchanged, so a subset computed from the schema that matches nothing
+no longer rewrites every column. The new
+{py:meth}`~datafusion.dataframe.DataFrame.fill_nan` behaves the same way.
+
+```python
+df.fill_null(0, subset=[]) # before: fills all columns; after: fills none
+df.fill_null(0) # fills all columns, before and after
+```
+
+### `spark.last_day` renamed its parameter
+
+The parameter of {py:func}`datafusion.functions.spark.last_day` is now named
+`date`, matching `pyspark.sql.functions.last_day`. Positional calls are
+unaffected; update any call passing it by keyword.
+
+```python
+spark.last_day(col=d) # before
+spark.last_day(date=d) # after
+```
+
### Changes to the `datafusion-python-util` crate
Extension libraries written in Rust usually depend on the
diff --git a/autoapi/datafusion/catalog/index.html b/autoapi/datafusion/catalog/index.html
index 61e127b0..d2d6a2ef 100644
--- a/autoapi/datafusion/catalog/index.html
+++ b/autoapi/datafusion/catalog/index.html
@@ -555,6 +555,9 @@
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.catalog.Table" title="datafusion.catalog.Table"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Table</span></code></a></p></td>
<td><p>A DataFusion table.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.catalog.TableProviderFactory" title="datafusion.catalog.TableProviderFactory"><code class="xref py py-obj docutils literal notranslate"><span class="pre">TableProviderFactory</span></code></a></p></td>
+<td><p>Abstract class for defining a Python based Table Provider Factory.</p></td>
+</tr>
</tbody>
</table>
</div>
@@ -891,6 +894,19 @@ interface (advanced)</p></li>
</dd></dl>
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.catalog.TableProviderFactory">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.catalog.</span></span><span class="sig-name descname"><span class="pre">TableProviderFactory</span></span><a class="headerlink" href="#datafusion.catalog.TableProviderFactory" title="Link to this definition">#</a></dt>
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">abc.ABC</span></code></p>
+<p>Abstract class for defining a Python based Table Provider Factory.</p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.catalog.TableProviderFactory.create">
+<em class="property"><span class="pre">abstract</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">create</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">cmd</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion.expr.CreateExternalTable</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="si [...]
+<dd><p>Create a table using the <code class="xref py py-class docutils literal notranslate"><span class="pre">CreateExternalTable</span></code>.</p>
+</dd></dl>
+
+</dd></dl>
+
</section>
</section>
@@ -1007,6 +1023,10 @@ interface (advanced)</p></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.catalog.Table.schema"><code class="docutils literal notranslate"><span class="pre">Table.schema</span></code></a></li>
</ul>
</li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.catalog.TableProviderFactory"><code class="docutils literal notranslate"><span class="pre">TableProviderFactory</span></code></a><ul class="nav section-nav flex-column">
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.catalog.TableProviderFactory.create"><code class="docutils literal notranslate"><span class="pre">TableProviderFactory.create()</span></code></a></li>
+</ul>
+</li>
</ul>
</li>
</ul>
diff --git a/autoapi/datafusion/context/index.html b/autoapi/datafusion/context/index.html
index d30c7b60..1e223b6d 100644
--- a/autoapi/datafusion/context/index.html
+++ b/autoapi/datafusion/context/index.html
@@ -2365,7 +2365,7 @@ this context.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.register_table_factory">
-<span class="sig-name descname"><span class="pre">register_table_factory</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">format</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">factory</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n">< [...]
+<span class="sig-name descname"><span class="pre">register_table_factory</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">format</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">factory</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n">< [...]
<dd><p>Register a <a class="reference internal" href="../index.html#datafusion.TableProviderFactoryExportable" title="datafusion.TableProviderFactoryExportable"><code class="xref py py-class docutils literal notranslate"><span class="pre">TableProviderFactoryExportable</span></code></a>.</p>
<p>The registered factory can be referenced from SQL DDL statements executed
against this context.</p>
@@ -2464,7 +2464,7 @@ against this context.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.set_query_planner">
-<span class="sig-name descname"><span class="pre">set_query_planner</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">planner</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../extensions/index.html#datafusion.extensions.QueryPlannerExportable" title="datafusion.extensions.QueryPlannerExportable"><span class="pre">datafusion.extensions.QueryPlannerE [...]
+<span class="sig-name descname"><span class="pre">set_query_planner</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">planner</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../extensions/index.html#datafusion.extensions.QueryPlannerExportable" title="datafusion.extensions.QueryPlannerExportable"><span class="pre">datafusion.extensions.QueryPlannerE [...]
<dd><p>Install a custom query planner on this session.</p>
<p>The planner is imported through its <code class="docutils literal notranslate"><span class="pre">__datafusion_query_planner__</span></code>
PyCapsule, in the same way
@@ -2827,7 +2827,7 @@ verbatim by <code class="docutils literal notranslate"><span class="pre">test_wi
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.with_logical_extension_codec">
-<span class="sig-name descname"><span class="pre">with_logical_extension_codec</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">codec</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../user_defined/index.html#datafusion.user_defined.LogicalExtensionCodecExportable" title="datafusion.user_defined.LogicalExtensionCodecExportable"><span class="pre">da [...]
+<span class="sig-name descname"><span class="pre">with_logical_extension_codec</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">codec</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../user_defined/index.html#datafusion.user_defined.LogicalExtensionCodecExportable" title="datafusion.user_defined.LogicalExtensionCodecExportable"><span class="pre">da [...]
<dd><p>Create a new session context with an additional logical codec.</p>
<p>Only FFI codecs are supported. Pass any object implementing
<code class="docutils literal notranslate"><span class="pre">__datafusion_logical_extension_codec__</span></code> (see
@@ -2891,7 +2891,7 @@ fine only if its plans never leave this session:</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.with_physical_extension_codec">
-<span class="sig-name descname"><span class="pre">with_physical_extension_codec</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">codec</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../user_defined/index.html#datafusion.user_defined.PhysicalExtensionCodecExportable" title="datafusion.user_defined.PhysicalExtensionCodecExportable"><span class="pre" [...]
+<span class="sig-name descname"><span class="pre">with_physical_extension_codec</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">codec</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../user_defined/index.html#datafusion.user_defined.PhysicalExtensionCodecExportable" title="datafusion.user_defined.PhysicalExtensionCodecExportable"><span class="pre" [...]
<dd><p>Create a new session context with an additional physical codec.</p>
<p>Only FFI codecs are supported. Pass any object implementing
<code class="docutils literal notranslate"><span class="pre">__datafusion_physical_extension_codec__</span></code> (see
diff --git a/autoapi/datafusion/dataframe/index.html b/autoapi/datafusion/dataframe/index.html
index b3c028c7..c0e7f292 100644
--- a/autoapi/datafusion/dataframe/index.html
+++ b/autoapi/datafusion/dataframe/index.html
@@ -566,9 +566,15 @@ overview of the execution model.</p>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.dataframe.DataFrameWriteOptions" title="datafusion.dataframe.DataFrameWriteOptions"><code class="xref py py-obj docutils literal notranslate"><span class="pre">DataFrameWriteOptions</span></code></a></p></td>
<td><p>Writer options for DataFrame.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.dataframe.ExplainFormat" title="datafusion.dataframe.ExplainFormat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ExplainFormat</span></code></a></p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.dataframe.ExplainAnalyzeLevel" title="datafusion.dataframe.ExplainAnalyzeLevel"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ExplainAnalyzeLevel</span></code></a></p></td>
+<td><p>Which metrics <a class="reference internal" href="#datafusion.dataframe.DataFrame.explain" title="datafusion.dataframe.DataFrame.explain"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.explain()</span></code></a> reports when <code class="docutils literal notranslate"><span class="pre">analyze=True</span></code>.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.dataframe.ExplainFormat" title="datafusion.dataframe.ExplainFormat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ExplainFormat</span></code></a></p></td>
<td><p>Output format for explain plans.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.dataframe.ExplainMetricCategory" title="datafusion.dataframe.ExplainMetricCategory"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ExplainMetricCategory</span></code></a></p></td>
+<td><p>Category of metric reported by <a class="reference internal" href="#datafusion.dataframe.DataFrame.explain" title="datafusion.dataframe.DataFrame.explain"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.explain()</span></code></a> with <code class="docutils literal notranslate"><span class="pre">analyze</span></code>.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.dataframe.InsertOp" title="datafusion.dataframe.InsertOp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">InsertOp</span></code></a></p></td>
<td><p>Insert operation mode.</p></td>
</tr>
@@ -1109,7 +1115,7 @@ if the original name is not strictly lower case.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.explain">
-<span class="sig-name descname"><span class="pre">explain</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">verbose</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-par [...]
+<span class="sig-name descname"><span class="pre">explain</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">verbose</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-par [...]
<dd><p>Print an explanation of the DataFrame’s plan so far.</p>
<p>If <code class="docutils literal notranslate"><span class="pre">analyze</span></code> is specified, runs the plan and reports metrics.</p>
<dl class="field-list simple">
@@ -1119,8 +1125,20 @@ if the original name is not strictly lower case.</p>
<li><p><strong>analyze</strong> – If <code class="docutils literal notranslate"><span class="pre">True</span></code>, the plan will run and metrics reported.</p></li>
<li><p><strong>format</strong> – Output format for the plan. Defaults to
<a class="reference internal" href="#datafusion.dataframe.ExplainFormat.INDENT" title="datafusion.dataframe.ExplainFormat.INDENT"><code class="xref py py-attr docutils literal notranslate"><span class="pre">ExplainFormat.INDENT</span></code></a>.</p></li>
+<li><p><strong>show_statistics</strong> – If <code class="docutils literal notranslate"><span class="pre">True</span></code>, include each operator’s statistics.
+<code class="docutils literal notranslate"><span class="pre">None</span></code> uses the <code class="docutils literal notranslate"><span class="pre">datafusion.explain.show_statistics</span></code>
+setting.</p></li>
+<li><p><strong>analyze_level</strong> – Which metrics to report with <code class="docutils literal notranslate"><span class="pre">analyze</span></code>. <code class="docutils literal notranslate"><span class="pre">None</span></code>
+uses the <code class="docutils literal notranslate"><span class="pre">datafusion.explain.analyze_level</span></code> setting.</p></li>
+<li><p><strong>analyze_categories</strong> – Report only metrics in these categories with
+<code class="docutils literal notranslate"><span class="pre">analyze</span></code>; an empty iterable reports none. <code class="docutils literal notranslate"><span class="pre">None</span></code> uses the
+<code class="docutils literal notranslate"><span class="pre">datafusion.explain.analyze_categories</span></code> setting.</p></li>
</ul>
</dd>
+<dt class="field-even">Raises<span class="colon">:</span></dt>
+<dd class="field-even"><p><strong>ValueError</strong> – If <code class="docutils literal notranslate"><span class="pre">show_statistics</span></code> is set with <code class="docutils literal notranslate"><span class="pre">analyze</span></code>, or
+<code class="docutils literal notranslate"><span class="pre">analyze_level</span></code> or <code class="docutils literal notranslate"><span class="pre">analyze_categories</span></code> is set without it.</p>
+</dd>
</dl>
<p class="rubric">Examples</p>
<p>Show the plan in tree format:</p>
@@ -1134,6 +1152,52 @@ if the original name is not strictly lower case.</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">explain</span><span class="p">(</span><span class="n">analyze</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
</div>
+<p>Show only row-count metrics:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion.dataframe</span><span class="w"> </span><span class="kn">import</span> <span class="n">ExplainMetricCategory</span>
+<span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">explain</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">analyze</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">analyze_categories</span><span class="o">=</span><span class="p">[</span><span class="n">ExplainMetricCategory</span><span class="o">.</span><span class="n">ROWS</span><span class="p">]</span>
+<span class="gp">... </span><span class="p">)</span>
+</pre></div>
+</div>
+</dd></dl>
+
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.fill_nan">
+<span class="sig-name descname"><span class="pre">fill_nan</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">float</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">subset</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pr [...]
+<dd><p>Fill NaN values in floating-point columns with a value.</p>
+<p>Only floating-point columns are changed; others are kept unchanged, as is
+any column <code class="docutils literal notranslate"><span class="pre">value</span></code> cannot be cast to. NaN is distinct from null, which
+<a class="reference internal" href="#datafusion.dataframe.DataFrame.fill_null" title="datafusion.dataframe.DataFrame.fill_null"><code class="xref py py-meth docutils literal notranslate"><span class="pre">fill_null()</span></code></a> handles. Fails on a DataFrame with an uppercase or
+dotted column name; see <a class="reference internal" href="../../../user-guide/common-operations/functions.html#fill-column-names"><span class="std std-ref">Column names with uppercase letters or dots</span></a>.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>value</strong> – Value to replace NaN with. Will be cast to match column type.</p></li>
+<li><p><strong>subset</strong> – Optional list of column names to fill. If None, fills all
+floating-point columns; an empty list fills none.</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns<span class="colon">:</span></dt>
+<dd class="field-even"><p>DataFrame with NaN values replaced.</p>
+</dd>
+</dl>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">SessionContext</span>
+<span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">nan</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="s2">"nan"</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="n">nan</span><span class="p">,</span> <span class="kc">None</span><span class="p">],</span> <span class="s2">"b"</span><span class="p" [...]
+<span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">)</span><span class="o">.</span><span class="n">to_pydict</span><span class="p">()</span>
+<span class="go">{'a': [1.0, 0.0, None], 'b': [0.0, 2.0, 3.0]}</span>
+</pre></div>
+</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">subset</span><span class="o">=</span><span class="p">[</span><span class="s2">"a"</span><span class="p">])</span><span class="o">.</span><span class="n">collect_column</span><span c [...]
+<span class="go">nan</span>
+</pre></div>
+</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">subset</span><span class="o">=</span><span class="p">[])</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"b"</span><span [...]
+<span class="go">nan</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py method">
@@ -1144,7 +1208,8 @@ if the original name is not strictly lower case.</p>
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>value</strong> – Value to replace nulls with. Will be cast to match column type.</p></li>
-<li><p><strong>subset</strong> – Optional list of column names to fill. If None, fills all columns.</p></li>
+<li><p><strong>subset</strong> – Optional list of column names to fill. If None, fills all columns;
+an empty list fills none.</p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
@@ -1160,11 +1225,17 @@ if the original name is not strictly lower case.</p>
<span class="go">[0, 1, 3]</span>
</pre></div>
</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">fill_null</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">subset</span><span class="o">=</span><span class="p">[])</span><span class="o">.</span><span class="n">to_pydict</span><span class="p">()</span>
+<span class="go">{'a': [1, None, 3], 'b': [None, 5, 6]}</span>
+</pre></div>
+</div>
<p class="rubric">Notes</p>
<ul class="simple">
<li><p>Only fills nulls in columns where the value can be cast to the column type</p></li>
<li><p>For columns where casting fails, the original column is kept unchanged</p></li>
<li><p>For columns not in subset, the original column is kept unchanged</p></li>
+<li><p>Fails on a DataFrame with an uppercase or dotted column name; see
+<a class="reference internal" href="../../../user-guide/common-operations/functions.html#fill-column-names"><span class="std std-ref">Column names with uppercase letters or dots</span></a></p></li>
</ul>
</dd></dl>
@@ -2074,6 +2145,25 @@ See the individual implementation and documentation for details.</p>
</dd></dl>
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainAnalyzeLevel">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.dataframe.</span></span><span class="sig-name descname"><span class="pre">ExplainAnalyzeLevel</span></span><a class="headerlink" href="#datafusion.dataframe.ExplainAnalyzeLevel" title="Link to this definition">#</a></dt>
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">enum.Enum</span></code></p>
+<p>Which metrics <a class="reference internal" href="#datafusion.dataframe.DataFrame.explain" title="datafusion.dataframe.DataFrame.explain"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.explain()</span></code></a> reports when <code class="docutils literal notranslate"><span class="pre">analyze=True</span></code>.</p>
+<dl class="py attribute">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainAnalyzeLevel.DEV">
+<span class="sig-name descname"><span class="pre">DEV</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'dev'</span></em><a class="headerlink" href="#datafusion.dataframe.ExplainAnalyzeLevel.DEV" title="Link to this definition">#</a></dt>
+<dd><p>All metrics, including those for deep operator-level introspection.</p>
+</dd></dl>
+
+<dl class="py attribute">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainAnalyzeLevel.SUMMARY">
+<span class="sig-name descname"><span class="pre">SUMMARY</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'summary'</span></em><a class="headerlink" href="#datafusion.dataframe.ExplainAnalyzeLevel.SUMMARY" title="Link to this definition">#</a></dt>
+<dd><p>Common metrics for finding which operator is slow.</p>
+</dd></dl>
+
+</dd></dl>
+
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.dataframe.ExplainFormat">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.dataframe.</span></span><span class="sig-name descname"><span class="pre">ExplainFormat</span></span><a class="headerlink" href="#datafusion.dataframe.ExplainFormat" title="Link to this definition">#</a></dt>
@@ -2106,6 +2196,37 @@ See the individual implementation and documentation for details.</p>
</dd></dl>
+<dl class="py class">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainMetricCategory">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.dataframe.</span></span><span class="sig-name descname"><span class="pre">ExplainMetricCategory</span></span><a class="headerlink" href="#datafusion.dataframe.ExplainMetricCategory" title="Link to this definition">#</a></dt>
+<dd><p>Bases: <code class="xref py py-obj docutils literal notranslate"><span class="pre">enum.Enum</span></code></p>
+<p>Category of metric reported by <a class="reference internal" href="#datafusion.dataframe.DataFrame.explain" title="datafusion.dataframe.DataFrame.explain"><code class="xref py py-meth docutils literal notranslate"><span class="pre">DataFrame.explain()</span></code></a> with <code class="docutils literal notranslate"><span class="pre">analyze</span></code>.</p>
+<dl class="py attribute">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainMetricCategory.BYTES">
+<span class="sig-name descname"><span class="pre">BYTES</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'bytes'</span></em><a class="headerlink" href="#datafusion.dataframe.ExplainMetricCategory.BYTES" title="Link to this definition">#</a></dt>
+<dd><p>Byte sizes, such as <code class="docutils literal notranslate"><span class="pre">output_bytes</span></code>.</p>
+</dd></dl>
+
+<dl class="py attribute">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainMetricCategory.ROWS">
+<span class="sig-name descname"><span class="pre">ROWS</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'rows'</span></em><a class="headerlink" href="#datafusion.dataframe.ExplainMetricCategory.ROWS" title="Link to this definition">#</a></dt>
+<dd><p>Row counts, such as <code class="docutils literal notranslate"><span class="pre">output_rows</span></code>.</p>
+</dd></dl>
+
+<dl class="py attribute">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainMetricCategory.TIMING">
+<span class="sig-name descname"><span class="pre">TIMING</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'timing'</span></em><a class="headerlink" href="#datafusion.dataframe.ExplainMetricCategory.TIMING" title="Link to this definition">#</a></dt>
+<dd><p>Elapsed times, such as <code class="docutils literal notranslate"><span class="pre">elapsed_compute</span></code>.</p>
+</dd></dl>
+
+<dl class="py attribute">
+<dt class="sig sig-object py" id="datafusion.dataframe.ExplainMetricCategory.UNCATEGORIZED">
+<span class="sig-name descname"><span class="pre">UNCATEGORIZED</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">'uncategorized'</span></em><a class="headerlink" href="#datafusion.dataframe.ExplainMetricCategory.UNCATEGORIZED" title="Link to this definition">#</a></dt>
+<dd><p>Metrics that declare no category.</p>
+</dd></dl>
+
+</dd></dl>
+
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.dataframe.InsertOp">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.dataframe.</span></span><span class="sig-name descname"><span class="pre">InsertOp</span></span><a class="headerlink" href="#datafusion.dataframe.InsertOp" title="Link to this definition">#</a></dt>
@@ -2476,6 +2597,7 @@ here.</p></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.execute_stream_partitioned"><code class="docutils literal notranslate"><span class="pre">DataFrame.execute_stream_partitioned()</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.execution_plan"><code class="docutils literal notranslate"><span class="pre">DataFrame.execution_plan()</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.explain"><code class="docutils literal notranslate"><span class="pre">DataFrame.explain()</span></code></a></li>
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.fill_nan"><code class="docutils literal notranslate"><span class="pre">DataFrame.fill_nan()</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.fill_null"><code class="docutils literal notranslate"><span class="pre">DataFrame.fill_null()</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.filter"><code class="docutils literal notranslate"><span class="pre">DataFrame.filter()</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.find_qualified_columns"><code class="docutils literal notranslate"><span class="pre">DataFrame.find_qualified_columns()</span></code></a></li>
@@ -2523,6 +2645,11 @@ here.</p></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.DataFrameWriteOptions._raw_write_options"><code class="docutils literal notranslate"><span class="pre">DataFrameWriteOptions._raw_write_options</span></code></a></li>
</ul>
</li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainAnalyzeLevel"><code class="docutils literal notranslate"><span class="pre">ExplainAnalyzeLevel</span></code></a><ul class="nav section-nav flex-column">
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainAnalyzeLevel.DEV"><code class="docutils literal notranslate"><span class="pre">ExplainAnalyzeLevel.DEV</span></code></a></li>
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainAnalyzeLevel.SUMMARY"><code class="docutils literal notranslate"><span class="pre">ExplainAnalyzeLevel.SUMMARY</span></code></a></li>
+</ul>
+</li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainFormat"><code class="docutils literal notranslate"><span class="pre">ExplainFormat</span></code></a><ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainFormat.GRAPHVIZ"><code class="docutils literal notranslate"><span class="pre">ExplainFormat.GRAPHVIZ</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainFormat.INDENT"><code class="docutils literal notranslate"><span class="pre">ExplainFormat.INDENT</span></code></a></li>
@@ -2530,6 +2657,13 @@ here.</p></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainFormat.TREE"><code class="docutils literal notranslate"><span class="pre">ExplainFormat.TREE</span></code></a></li>
</ul>
</li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainMetricCategory"><code class="docutils literal notranslate"><span class="pre">ExplainMetricCategory</span></code></a><ul class="nav section-nav flex-column">
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainMetricCategory.BYTES"><code class="docutils literal notranslate"><span class="pre">ExplainMetricCategory.BYTES</span></code></a></li>
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainMetricCategory.ROWS"><code class="docutils literal notranslate"><span class="pre">ExplainMetricCategory.ROWS</span></code></a></li>
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainMetricCategory.TIMING"><code class="docutils literal notranslate"><span class="pre">ExplainMetricCategory.TIMING</span></code></a></li>
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.ExplainMetricCategory.UNCATEGORIZED"><code class="docutils literal notranslate"><span class="pre">ExplainMetricCategory.UNCATEGORIZED</span></code></a></li>
+</ul>
+</li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.InsertOp"><code class="docutils literal notranslate"><span class="pre">InsertOp</span></code></a><ul class="nav section-nav flex-column">
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.InsertOp.APPEND"><code class="docutils literal notranslate"><span class="pre">InsertOp.APPEND</span></code></a></li>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.dataframe.InsertOp.OVERWRITE"><code class="docutils literal notranslate"><span class="pre">InsertOp.OVERWRITE</span></code></a></li>
diff --git a/autoapi/datafusion/expr/index.html b/autoapi/datafusion/expr/index.html
index 3839f4d9..935da77b 100644
--- a/autoapi/datafusion/expr/index.html
+++ b/autoapi/datafusion/expr/index.html
@@ -1339,8 +1339,9 @@ through <code class="docutils literal notranslate"><span class="pre">pickle.dump
<span class="sig-name descname"><span class="pre">distinct</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">ExprFuncBuilder</span></span></span><a class="headerlink" href="#datafusion.expr.Expr.distinct" title="Link to this definition">#</a></dt>
<dd><p>Only evaluate distinct values for an aggregate function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on a window
+function unless that is an aggregate used as one. If used on any other type of
+expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py method">
@@ -1378,8 +1379,9 @@ type of expression, an error will be generated when <code class="docutils litera
<span class="sig-name descname"><span class="pre">filter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">filter</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">Expr</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon"> [...]
<dd><p>Filter an aggregate function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on a window
+function unless that is an aggregate used as one. If used on any other type of
+expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py method">
@@ -1604,10 +1606,15 @@ type of expression, an error will be generated when <code class="docutils litera
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.expr.Expr.over">
<span class="sig-name descname"><span class="pre">over</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">window</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.expr.Window" title="datafusion.expr.Window"><span class="pre">Window</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon"> [...]
-<dd><p>Turn an aggregate function into a window function.</p>
-<p>This function turns any aggregate function into a window function. With the
+<dd><p>Evaluate an aggregate or window function over a window.</p>
+<p>On an aggregate function this turns it into a window function. With the
exception of <code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, how each of the parameters is used is determined
by the underlying aggregate function.</p>
+<p>On an aggregate, the <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, <code class="docutils literal notranslate"><span class="pre">filter</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code> options
+it was built with are kept, and an <code class="docutils literal notranslate"><span class="pre">order_by</span></code> raises; see
+<a class="reference internal" href="../../../user-guide/common-operations/windows.html#aggregate-over-options"><span class="std std-ref">Options set on the aggregate</span></a>.</p>
+<p>On a window function that already has a <code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, or
+<code class="docutils literal notranslate"><span class="pre">window_frame</span></code>, this raises; see <a class="reference internal" href="../../../user-guide/common-operations/windows.html#window-function-chaining"><span class="std std-ref">Chaining onto a window function</span></a>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>window</strong> – Window definition</p>
@@ -1620,8 +1627,10 @@ by the underlying aggregate function.</p>
<span class="sig-name descname"><span class="pre">partition_by</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">partition_by</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">Expr</span></a></span></em><span class="sig-paren">)</span [...]
<dd><p>Set the partitioning for a window function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on an
+aggregate function that has not been turned into a window function with
+<a class="reference internal" href="#datafusion.expr.Expr.over" title="datafusion.expr.Expr.over"><code class="xref py py-meth docutils literal notranslate"><span class="pre">over()</span></code></a>. If used on any other type of expression, an error will be
+generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py method">
@@ -1912,10 +1921,12 @@ SqlType Enum which this expression represents.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.expr.Expr.window_frame">
<span class="sig-name descname"><span class="pre">window_frame</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">window_frame</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.expr.WindowFrame" title="datafusion.expr.WindowFrame"><span class="pre">WindowFrame</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return">< [...]
-<dd><p>Set the frame fora window function.</p>
+<dd><p>Set the frame for a window function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on an
+aggregate function that has not been turned into a window function with
+<a class="reference internal" href="#datafusion.expr.Expr.over" title="datafusion.expr.Expr.over"><code class="xref py py-meth docutils literal notranslate"><span class="pre">over()</span></code></a>. If used on any other type of expression, an error will be
+generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py attribute">
diff --git a/autoapi/datafusion/extensions/index.html b/autoapi/datafusion/extensions/index.html
index a38a90d3..be17c736 100644
--- a/autoapi/datafusion/extensions/index.html
+++ b/autoapi/datafusion/extensions/index.html
@@ -859,7 +859,7 @@ of them is the no-op:</p>
</div>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.extensions.SessionPlannerExportable.__datafusion_session_planner__">
-<span class="sig-name descname"><span class="pre">__datafusion_session_planner__</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">ctx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><span class="pre">datafusion.context.SessionContext</span></a></span> [...]
+<span class="sig-name descname"><span class="pre">__datafusion_session_planner__</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">ctx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><span class="pre">datafusion.context.SessionContext</span></a></span> [...]
<dd></dd></dl>
</dd></dl>
diff --git a/autoapi/datafusion/functions/index.html b/autoapi/datafusion/functions/index.html
index 2dd48c62..27c067a6 100644
--- a/autoapi/datafusion/functions/index.html
+++ b/autoapi/datafusion/functions/index.html
@@ -586,21 +586,27 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.any_match" title="datafusion.functions.any_match"><code class="xref py py-obj docutils literal notranslate"><span class="pre">any_match</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> if any element of an array satisfies a predicate.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.approx_distinct" title="datafusion.functions.approx_distinct"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_distinct</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.any_value" title="datafusion.functions.any_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">any_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns an arbitrary non-null value from each group.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.approx_distinct" title="datafusion.functions.approx_distinct"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_distinct</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the approximate number of distinct values.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.approx_median" title="datafusion.functions.approx_median"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_median</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.approx_median" title="datafusion.functions.approx_median"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_median</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the approximate median value.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.approx_percentile_cont" title="datafusion.functions.approx_percentile_cont"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_percentile_cont</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.approx_percentile_cont" title="datafusion.functions.approx_percentile_cont"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_percentile_cont</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the value that is approximately at a given percentile of <code class="docutils literal notranslate"><span class="pre">expr</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.approx_percentile_cont_with_weight" title="datafusion.functions.approx_percentile_cont_with_weight"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_percentile_cont_with_weight</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.approx_percentile_cont_with_weight" title="datafusion.functions.approx_percentile_cont_with_weight"><code class="xref py py-obj docutils literal notranslate"><span class="pre">approx_percentile_cont_with_weight</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the value of the weighted approximate percentile.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array" title="datafusion.functions.array"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array" title="datafusion.functions.array"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array using the specified input expressions.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_add" title="datafusion.functions.array_add"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_add</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the element-wise sum of two numeric arrays of equal length.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_agg" title="datafusion.functions.array_agg"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_agg</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Aggregate values into an array.</p></td>
</tr>
@@ -613,42 +619,48 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_append" title="datafusion.functions.array_append"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_append</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Appends an element to the end of an array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_cat" title="datafusion.functions.array_cat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_cat</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_avg" title="datafusion.functions.array_avg"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_avg</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the arithmetic mean of the elements of a numeric array.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_cat" title="datafusion.functions.array_cat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_cat</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Concatenates the input arrays.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_compact" title="datafusion.functions.array_compact"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_compact</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_compact" title="datafusion.functions.array_compact"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_compact</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Removes NULL values from the array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_concat" title="datafusion.functions.array_concat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_concat</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_concat" title="datafusion.functions.array_concat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_concat</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Concatenates the input arrays.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_contains" title="datafusion.functions.array_contains"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_contains</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_contains" title="datafusion.functions.array_contains"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_contains</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns true if the element appears in the array, otherwise false.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_dims" title="datafusion.functions.array_dims"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_dims</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_dims" title="datafusion.functions.array_dims"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_dims</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array of the array's dimensions.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_distance" title="datafusion.functions.array_distance"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_distance</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_distance" title="datafusion.functions.array_distance"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_distance</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the Euclidean distance between two numeric arrays.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_distinct" title="datafusion.functions.array_distinct"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_distinct</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_distinct" title="datafusion.functions.array_distinct"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_distinct</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns distinct values from the array after removing duplicates.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_element" title="datafusion.functions.array_element"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_element</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_element" title="datafusion.functions.array_element"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_element</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Extracts the element with the index n from the array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_empty" title="datafusion.functions.array_empty"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_empty</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_empty" title="datafusion.functions.array_empty"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_empty</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a boolean indicating whether the array is empty.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_except" title="datafusion.functions.array_except"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_except</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_except" title="datafusion.functions.array_except"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_except</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the elements that appear in <code class="docutils literal notranslate"><span class="pre">array1</span></code> but not in <code class="docutils literal notranslate"><span class="pre">array2</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_extract" title="datafusion.functions.array_extract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_extract</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_extract" title="datafusion.functions.array_extract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_extract</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Extracts the element with the index n from the array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_filter" title="datafusion.functions.array_filter"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_filter</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_filter" title="datafusion.functions.array_filter"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_filter</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Keep the elements of <code class="docutils literal notranslate"><span class="pre">array</span></code> for which <code class="docutils literal notranslate"><span class="pre">predicate</span></code> is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_first" title="datafusion.functions.array_first"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_first</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Return the first element of <code class="docutils literal notranslate"><span class="pre">array</span></code> for which <code class="docutils literal notranslate"><span class="pre">predicate</span></code> is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_has" title="datafusion.functions.array_has"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_has</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns true if the element appears in the first array, otherwise false.</p></td>
</tr>
@@ -697,45 +709,57 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_prepend" title="datafusion.functions.array_prepend"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_prepend</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Prepends an element to the beginning of an array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_push_back" title="datafusion.functions.array_push_back"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_push_back</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_product" title="datafusion.functions.array_product"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_product</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the product of the elements of a numeric array.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_push_back" title="datafusion.functions.array_push_back"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_push_back</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Appends an element to the end of an array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_push_front" title="datafusion.functions.array_push_front"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_push_front</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_push_front" title="datafusion.functions.array_push_front"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_push_front</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Prepends an element to the beginning of an array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_remove" title="datafusion.functions.array_remove"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_remove</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_remove" title="datafusion.functions.array_remove"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_remove</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Removes the first element from the array equal to the given value.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_remove_all" title="datafusion.functions.array_remove_all"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_remove_all</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_remove_all" title="datafusion.functions.array_remove_all"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_remove_all</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Removes all elements from the array equal to the given value.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_remove_n" title="datafusion.functions.array_remove_n"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_remove_n</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_remove_n" title="datafusion.functions.array_remove_n"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_remove_n</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Removes the first <code class="docutils literal notranslate"><span class="pre">max</span></code> elements from the array equal to the given value.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_repeat" title="datafusion.functions.array_repeat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_repeat</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_repeat" title="datafusion.functions.array_repeat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_repeat</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array containing <code class="docutils literal notranslate"><span class="pre">element</span></code> <code class="docutils literal notranslate"><span class="pre">count</span></code> times.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_replace" title="datafusion.functions.array_replace"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_replace</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_replace" title="datafusion.functions.array_replace"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_replace</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Replaces the first occurrence of <code class="docutils literal notranslate"><span class="pre">from_val</span></code> with <code class="docutils literal notranslate"><span class="pre">to_val</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_replace_all" title="datafusion.functions.array_replace_all"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_replace_all</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_replace_all" title="datafusion.functions.array_replace_all"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_replace_all</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Replaces all occurrences of <code class="docutils literal notranslate"><span class="pre">from_val</span></code> with <code class="docutils literal notranslate"><span class="pre">to_val</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_replace_n" title="datafusion.functions.array_replace_n"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_replace_n</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_replace_n" title="datafusion.functions.array_replace_n"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_replace_n</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Replace <code class="docutils literal notranslate"><span class="pre">n</span></code> occurrences of <code class="docutils literal notranslate"><span class="pre">from_val</span></code> with <code class="docutils literal notranslate"><span class="pre">to_val</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_resize" title="datafusion.functions.array_resize"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_resize</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_resize" title="datafusion.functions.array_resize"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_resize</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array with the specified size filled.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_reverse" title="datafusion.functions.array_reverse"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_reverse</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_reverse" title="datafusion.functions.array_reverse"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_reverse</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Reverses the order of elements in the array.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_scale" title="datafusion.functions.array_scale"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_scale</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Multiplies each element of a numeric array by <code class="docutils literal notranslate"><span class="pre">scalar</span></code>.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_slice" title="datafusion.functions.array_slice"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_slice</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a slice of the array.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_sort" title="datafusion.functions.array_sort"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_sort</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Sort an array.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_subtract" title="datafusion.functions.array_subtract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_subtract</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the element-wise difference of two numeric arrays of equal length.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.array_sum" title="datafusion.functions.array_sum"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_sum</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the sum of the elements of a numeric array.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.array_to_string" title="datafusion.functions.array_to_string"><code class="xref py py-obj docutils literal notranslate"><span class="pre">array_to_string</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts each element to its text representation.</p></td>
</tr>
@@ -806,7 +830,7 @@ for categorized catalogs of aggregate and window functions.</p>
<td><p>Computes the boolean OR of the argument.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.btrim" title="datafusion.functions.btrim"><code class="xref py py-obj docutils literal notranslate"><span class="pre">btrim</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Removes all characters, spaces by default, from both sides of a string.</p></td>
+<td><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from both sides of a string.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.cardinality" title="datafusion.functions.cardinality"><code class="xref py py-obj docutils literal notranslate"><span class="pre">cardinality</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the total number of elements in the array.</p></td>
@@ -943,51 +967,57 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.factorial" title="datafusion.functions.factorial"><code class="xref py py-obj docutils literal notranslate"><span class="pre">factorial</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the factorial of the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.find_in_set" title="datafusion.functions.find_in_set"><code class="xref py py-obj docutils literal notranslate"><span class="pre">find_in_set</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.file_row_index" title="datafusion.functions.file_row_index"><code class="xref py py-obj docutils literal notranslate"><span class="pre">file_row_index</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the zero-based position of the current row within its source file.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.find_in_set" title="datafusion.functions.find_in_set"><code class="xref py py-obj docutils literal notranslate"><span class="pre">find_in_set</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Find a string in a list of strings.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.first_value" title="datafusion.functions.first_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">first_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.first_value" title="datafusion.functions.first_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">first_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the first value in a group of values.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.flatten" title="datafusion.functions.flatten"><code class="xref py py-obj docutils literal notranslate"><span class="pre">flatten</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.flatten" title="datafusion.functions.flatten"><code class="xref py py-obj docutils literal notranslate"><span class="pre">flatten</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Flattens an array of arrays into a single array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.floor" title="datafusion.functions.floor"><code class="xref py py-obj docutils literal notranslate"><span class="pre">floor</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.floor" title="datafusion.functions.floor"><code class="xref py py-obj docutils literal notranslate"><span class="pre">floor</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the nearest integer less than or equal to the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.from_unixtime" title="datafusion.functions.from_unixtime"><code class="xref py py-obj docutils literal notranslate"><span class="pre">from_unixtime</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.from_unixtime" title="datafusion.functions.from_unixtime"><code class="xref py py-obj docutils literal notranslate"><span class="pre">from_unixtime</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts an integer to RFC3339 timestamp format string.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.gcd" title="datafusion.functions.gcd"><code class="xref py py-obj docutils literal notranslate"><span class="pre">gcd</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.gcd" title="datafusion.functions.gcd"><code class="xref py py-obj docutils literal notranslate"><span class="pre">gcd</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the greatest common divisor.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.gen_series" title="datafusion.functions.gen_series"><code class="xref py py-obj docutils literal notranslate"><span class="pre">gen_series</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Creates a list of values in the range between start and stop.</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.gen_series" title="datafusion.functions.gen_series"><code class="xref py py-obj docutils literal notranslate"><span class="pre">gen_series</span></code></a>(…)</p></td>
+<td><p>Creates a list of values from <code class="docutils literal notranslate"><span class="pre">start</span></code> up to and including <code class="docutils literal notranslate"><span class="pre">stop</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.generate_series" title="datafusion.functions.generate_series"><code class="xref py py-obj docutils literal notranslate"><span class="pre">generate_series</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Creates a list of values in the range between start and stop.</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.generate_series" title="datafusion.functions.generate_series"><code class="xref py py-obj docutils literal notranslate"><span class="pre">generate_series</span></code></a>(…)</p></td>
+<td><p>Creates a list of values from <code class="docutils literal notranslate"><span class="pre">start</span></code> up to and including <code class="docutils literal notranslate"><span class="pre">stop</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.get_field" title="datafusion.functions.get_field"><code class="xref py py-obj docutils literal notranslate"><span class="pre">get_field</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.get_field" title="datafusion.functions.get_field"><code class="xref py py-obj docutils literal notranslate"><span class="pre">get_field</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Extracts a (possibly nested) field from a struct or map by name.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.greatest" title="datafusion.functions.greatest"><code class="xref py py-obj docutils literal notranslate"><span class="pre">greatest</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.greatest" title="datafusion.functions.greatest"><code class="xref py py-obj docutils literal notranslate"><span class="pre">greatest</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the greatest value from a list of expressions.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.grouping" title="datafusion.functions.grouping"><code class="xref py py-obj docutils literal notranslate"><span class="pre">grouping</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.grouping" title="datafusion.functions.grouping"><code class="xref py py-obj docutils literal notranslate"><span class="pre">grouping</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Indicates whether a column is aggregated across in the current row.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.ifnull" title="datafusion.functions.ifnull"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ifnull</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.ifnull" title="datafusion.functions.ifnull"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ifnull</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns <code class="docutils literal notranslate"><span class="pre">x</span></code> if <code class="docutils literal notranslate"><span class="pre">x</span></code> is not NULL. Otherwise returns <code class="docutils literal notranslate"><span class="pre">y</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.in_list" title="datafusion.functions.in_list"><code class="xref py py-obj docutils literal notranslate"><span class="pre">in_list</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.in_list" title="datafusion.functions.in_list"><code class="xref py py-obj docutils literal notranslate"><span class="pre">in_list</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns whether the argument is contained within the list <code class="docutils literal notranslate"><span class="pre">values</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.initcap" title="datafusion.functions.initcap"><code class="xref py py-obj docutils literal notranslate"><span class="pre">initcap</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.initcap" title="datafusion.functions.initcap"><code class="xref py py-obj docutils literal notranslate"><span class="pre">initcap</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Set the initial letter of each word to capital.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.inner_product" title="datafusion.functions.inner_product"><code class="xref py py-obj docutils literal notranslate"><span class="pre">inner_product</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.inner_product" title="datafusion.functions.inner_product"><code class="xref py py-obj docutils literal notranslate"><span class="pre">inner_product</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the inner (dot) product of two numeric arrays.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.input_file_name" title="datafusion.functions.input_file_name"><code class="xref py py-obj docutils literal notranslate"><span class="pre">input_file_name</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the path of the file that produced the current row.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.instr" title="datafusion.functions.instr"><code class="xref py py-obj docutils literal notranslate"><span class="pre">instr</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Finds the position from where the <code class="docutils literal notranslate"><span class="pre">substring</span></code> matches the <code class="docutils literal notranslate"><span class="pre">string</span></code>.</p></td>
</tr>
@@ -1030,15 +1060,21 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.levenshtein" title="datafusion.functions.levenshtein"><code class="xref py py-obj docutils literal notranslate"><span class="pre">levenshtein</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the Levenshtein distance between the two given strings.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_any_match" title="datafusion.functions.list_any_match"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_any_match</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_add" title="datafusion.functions.list_add"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_add</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the element-wise sum of two numeric lists of equal length.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_any_match" title="datafusion.functions.list_any_match"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_any_match</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> if any element of a list satisfies a predicate.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_any_value" title="datafusion.functions.list_any_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_any_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_any_value" title="datafusion.functions.list_any_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_any_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the first non-null element in the array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_append" title="datafusion.functions.list_append"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_append</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_append" title="datafusion.functions.list_append"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_append</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Appends an element to the end of an array.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_avg" title="datafusion.functions.list_avg"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_avg</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the arithmetic mean of the elements of a numeric list.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_cat" title="datafusion.functions.list_cat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_cat</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Concatenates the input arrays.</p></td>
</tr>
@@ -1075,57 +1111,63 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_filter" title="datafusion.functions.list_filter"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_filter</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Keep the elements of a list for which a predicate is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_has" title="datafusion.functions.list_has"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_has</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_first" title="datafusion.functions.list_first"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_first</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Return the first element of a list for which a predicate is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_has" title="datafusion.functions.list_has"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_has</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns true if the element appears in the array, otherwise false.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_has_all" title="datafusion.functions.list_has_all"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_has_all</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_has_all" title="datafusion.functions.list_has_all"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_has_all</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Determines if there is complete overlap <code class="docutils literal notranslate"><span class="pre">second_array</span></code> in <code class="docutils literal notranslate"><span class="pre">first_array</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_has_any" title="datafusion.functions.list_has_any"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_has_any</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_has_any" title="datafusion.functions.list_has_any"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_has_any</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Determine if there is an overlap between <code class="docutils literal notranslate"><span class="pre">first_array</span></code> and <code class="docutils literal notranslate"><span class="pre">second_array</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_indexof" title="datafusion.functions.list_indexof"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_indexof</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_indexof" title="datafusion.functions.list_indexof"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_indexof</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Return the position of the first occurrence of <code class="docutils literal notranslate"><span class="pre">element</span></code> in <code class="docutils literal notranslate"><span class="pre">array</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_intersect" title="datafusion.functions.list_intersect"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_intersect</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_intersect" title="datafusion.functions.list_intersect"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_intersect</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an the intersection of <code class="docutils literal notranslate"><span class="pre">array1</span></code> and <code class="docutils literal notranslate"><span class="pre">array2</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_join" title="datafusion.functions.list_join"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_join</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_join" title="datafusion.functions.list_join"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_join</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts each element to its text representation.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_length" title="datafusion.functions.list_length"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_length</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_length" title="datafusion.functions.list_length"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_length</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the length of the array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_max" title="datafusion.functions.list_max"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_max</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_max" title="datafusion.functions.list_max"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_max</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the maximum value in the array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_min" title="datafusion.functions.list_min"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_min</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_min" title="datafusion.functions.list_min"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_min</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the minimum value in the array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_ndims" title="datafusion.functions.list_ndims"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_ndims</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_ndims" title="datafusion.functions.list_ndims"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_ndims</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the number of dimensions of the array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_normalize" title="datafusion.functions.list_normalize"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_normalize</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_normalize" title="datafusion.functions.list_normalize"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_normalize</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Scales a numeric array so it has Euclidean length 1.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_overlap" title="datafusion.functions.list_overlap"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_overlap</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_overlap" title="datafusion.functions.list_overlap"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_overlap</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns true if any element appears in both arrays.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_pop_back" title="datafusion.functions.list_pop_back"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_pop_back</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_pop_back" title="datafusion.functions.list_pop_back"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_pop_back</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the array without the last element.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_pop_front" title="datafusion.functions.list_pop_front"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_pop_front</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_pop_front" title="datafusion.functions.list_pop_front"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_pop_front</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the array without the first element.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_position" title="datafusion.functions.list_position"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_position</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_position" title="datafusion.functions.list_position"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_position</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Return the position of the first occurrence of <code class="docutils literal notranslate"><span class="pre">element</span></code> in <code class="docutils literal notranslate"><span class="pre">array</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_positions" title="datafusion.functions.list_positions"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_positions</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_positions" title="datafusion.functions.list_positions"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_positions</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Searches for an element in the array and returns all occurrences.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_prepend" title="datafusion.functions.list_prepend"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_prepend</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_prepend" title="datafusion.functions.list_prepend"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_prepend</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Prepends an element to the beginning of an array.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_product" title="datafusion.functions.list_product"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_product</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the product of the elements of a numeric list.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_push_back" title="datafusion.functions.list_push_back"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_push_back</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Appends an element to the end of an array.</p></td>
</tr>
@@ -1159,149 +1201,161 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_reverse" title="datafusion.functions.list_reverse"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_reverse</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Reverses the order of elements in the array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_slice" title="datafusion.functions.list_slice"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_slice</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_scale" title="datafusion.functions.list_scale"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_scale</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Multiplies each element of a numeric list by a scalar.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_slice" title="datafusion.functions.list_slice"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_slice</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a slice of the array.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_sort" title="datafusion.functions.list_sort"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_sort</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_sort" title="datafusion.functions.list_sort"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_sort</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Sorts the array.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_to_string" title="datafusion.functions.list_to_string"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_to_string</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_subtract" title="datafusion.functions.list_subtract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_subtract</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the element-wise difference of two numeric lists of equal length.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_sum" title="datafusion.functions.list_sum"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_sum</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns the sum of the elements of a numeric list.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_to_string" title="datafusion.functions.list_to_string"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_to_string</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts each element to its text representation.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_transform" title="datafusion.functions.list_transform"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_transform</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_transform" title="datafusion.functions.list_transform"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_transform</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Transform each element of a list with a lambda.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_union" title="datafusion.functions.list_union"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_union</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_union" title="datafusion.functions.list_union"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_union</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array of the elements in the union of array1 and array2.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.list_zip" title="datafusion.functions.list_zip"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_zip</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.list_zip" title="datafusion.functions.list_zip"><code class="xref py py-obj docutils literal notranslate"><span class="pre">list_zip</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Combines multiple arrays into a single array of structs.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.ln" title="datafusion.functions.ln"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ln</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.ln" title="datafusion.functions.ln"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ln</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the natural logarithm (base e) of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.log" title="datafusion.functions.log"><code class="xref py py-obj docutils literal notranslate"><span class="pre">log</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.log" title="datafusion.functions.log"><code class="xref py py-obj docutils literal notranslate"><span class="pre">log</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the logarithm of a number for a particular <code class="docutils literal notranslate"><span class="pre">base</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.log10" title="datafusion.functions.log10"><code class="xref py py-obj docutils literal notranslate"><span class="pre">log10</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.log10" title="datafusion.functions.log10"><code class="xref py py-obj docutils literal notranslate"><span class="pre">log10</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Base 10 logarithm of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.log2" title="datafusion.functions.log2"><code class="xref py py-obj docutils literal notranslate"><span class="pre">log2</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.log2" title="datafusion.functions.log2"><code class="xref py py-obj docutils literal notranslate"><span class="pre">log2</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Base 2 logarithm of the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.lower" title="datafusion.functions.lower"><code class="xref py py-obj docutils literal notranslate"><span class="pre">lower</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.lower" title="datafusion.functions.lower"><code class="xref py py-obj docutils literal notranslate"><span class="pre">lower</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string to lowercase.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.lpad" title="datafusion.functions.lpad"><code class="xref py py-obj docutils literal notranslate"><span class="pre">lpad</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.lpad" title="datafusion.functions.lpad"><code class="xref py py-obj docutils literal notranslate"><span class="pre">lpad</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Add left padding to a string.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.ltrim" title="datafusion.functions.ltrim"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ltrim</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Removes all characters, spaces by default, from the beginning of a string.</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.ltrim" title="datafusion.functions.ltrim"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ltrim</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from the beginning of a string.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.make_array" title="datafusion.functions.make_array"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_array</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.make_array" title="datafusion.functions.make_array"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_array</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array using the specified input expressions.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.make_date" title="datafusion.functions.make_date"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_date</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.make_date" title="datafusion.functions.make_date"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_date</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Make a date from year, month and day component parts.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.make_list" title="datafusion.functions.make_list"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_list</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.make_list" title="datafusion.functions.make_list"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_list</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an array using the specified input expressions.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.make_map" title="datafusion.functions.make_map"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_map</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.make_map" title="datafusion.functions.make_map"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_map</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a map expression.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.make_time" title="datafusion.functions.make_time"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_time</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.make_time" title="datafusion.functions.make_time"><code class="xref py py-obj docutils literal notranslate"><span class="pre">make_time</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Make a time from hour, minute and second component parts.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.map_entries" title="datafusion.functions.map_entries"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_entries</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.map_entries" title="datafusion.functions.map_entries"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_entries</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a list of all entries (key-value struct pairs) in the map.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.map_extract" title="datafusion.functions.map_extract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_extract</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.map_extract" title="datafusion.functions.map_extract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_extract</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the value for a given key in the map.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.map_keys" title="datafusion.functions.map_keys"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_keys</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.map_keys" title="datafusion.functions.map_keys"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_keys</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a list of all keys in the map.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.map_values" title="datafusion.functions.map_values"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_values</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.map_values" title="datafusion.functions.map_values"><code class="xref py py-obj docutils literal notranslate"><span class="pre">map_values</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a list of all values in the map.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.max" title="datafusion.functions.max"><code class="xref py py-obj docutils literal notranslate"><span class="pre">max</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.max" title="datafusion.functions.max"><code class="xref py py-obj docutils literal notranslate"><span class="pre">max</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Aggregate function that returns the maximum value of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.md5" title="datafusion.functions.md5"><code class="xref py py-obj docutils literal notranslate"><span class="pre">md5</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.md5" title="datafusion.functions.md5"><code class="xref py py-obj docutils literal notranslate"><span class="pre">md5</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes an MD5 128-bit checksum for a string expression.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.mean" title="datafusion.functions.mean"><code class="xref py py-obj docutils literal notranslate"><span class="pre">mean</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.mean" title="datafusion.functions.mean"><code class="xref py py-obj docutils literal notranslate"><span class="pre">mean</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the average (mean) value of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.median" title="datafusion.functions.median"><code class="xref py py-obj docutils literal notranslate"><span class="pre">median</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.median" title="datafusion.functions.median"><code class="xref py py-obj docutils literal notranslate"><span class="pre">median</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the median of a set of numbers.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.min" title="datafusion.functions.min"><code class="xref py py-obj docutils literal notranslate"><span class="pre">min</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.min" title="datafusion.functions.min"><code class="xref py py-obj docutils literal notranslate"><span class="pre">min</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Aggregate function that returns the minimum value of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.named_struct" title="datafusion.functions.named_struct"><code class="xref py py-obj docutils literal notranslate"><span class="pre">named_struct</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.named_struct" title="datafusion.functions.named_struct"><code class="xref py py-obj docutils literal notranslate"><span class="pre">named_struct</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a struct with the given names and arguments pairs.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.nanvl" title="datafusion.functions.nanvl"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nanvl</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.nanvl" title="datafusion.functions.nanvl"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nanvl</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns <code class="docutils literal notranslate"><span class="pre">x</span></code> if <code class="docutils literal notranslate"><span class="pre">x</span></code> is not <code class="docutils literal notranslate"><span class="pre">NaN</span></code>. Otherwise returns <code class="docutils literal notranslate"><span class="pre">y</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.now" title="datafusion.functions.now"><code class="xref py py-obj docutils literal notranslate"><span class="pre">now</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.now" title="datafusion.functions.now"><code class="xref py py-obj docutils literal notranslate"><span class="pre">now</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the current timestamp in nanoseconds.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.nth_value" title="datafusion.functions.nth_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nth_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.nth_value" title="datafusion.functions.nth_value"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nth_value</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the n-th value in a group of values.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.ntile" title="datafusion.functions.ntile"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ntile</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.ntile" title="datafusion.functions.ntile"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ntile</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Create a n-tile window function.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.nullif" title="datafusion.functions.nullif"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nullif</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.nullif" title="datafusion.functions.nullif"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nullif</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns NULL if expr1 equals expr2; otherwise it returns expr1.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.nvl" title="datafusion.functions.nvl"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nvl</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.nvl" title="datafusion.functions.nvl"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nvl</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns <code class="docutils literal notranslate"><span class="pre">x</span></code> if <code class="docutils literal notranslate"><span class="pre">x</span></code> is not <code class="docutils literal notranslate"><span class="pre">NULL</span></code>. Otherwise returns <code class="docutils literal notranslate"><span class="pre">y</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.nvl2" title="datafusion.functions.nvl2"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nvl2</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.nvl2" title="datafusion.functions.nvl2"><code class="xref py py-obj docutils literal notranslate"><span class="pre">nvl2</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns <code class="docutils literal notranslate"><span class="pre">y</span></code> if <code class="docutils literal notranslate"><span class="pre">x</span></code> is not NULL. Otherwise returns <code class="docutils literal notranslate"><span class="pre">z</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.octet_length" title="datafusion.functions.octet_length"><code class="xref py py-obj docutils literal notranslate"><span class="pre">octet_length</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.octet_length" title="datafusion.functions.octet_length"><code class="xref py py-obj docutils literal notranslate"><span class="pre">octet_length</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the number of bytes of a string.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.order_by" title="datafusion.functions.order_by"><code class="xref py py-obj docutils literal notranslate"><span class="pre">order_by</span></code></a>(→ datafusion.expr.SortExpr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.order_by" title="datafusion.functions.order_by"><code class="xref py py-obj docutils literal notranslate"><span class="pre">order_by</span></code></a>(→ datafusion.expr.SortExpr)</p></td>
<td><p>Creates a new sort expression.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.overlay" title="datafusion.functions.overlay"><code class="xref py py-obj docutils literal notranslate"><span class="pre">overlay</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.overlay" title="datafusion.functions.overlay"><code class="xref py py-obj docutils literal notranslate"><span class="pre">overlay</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Replace a substring with a new substring.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.percent_rank" title="datafusion.functions.percent_rank"><code class="xref py py-obj docutils literal notranslate"><span class="pre">percent_rank</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.percent_rank" title="datafusion.functions.percent_rank"><code class="xref py py-obj docutils literal notranslate"><span class="pre">percent_rank</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Create a percent_rank window function.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.percentile_cont" title="datafusion.functions.percentile_cont"><code class="xref py py-obj docutils literal notranslate"><span class="pre">percentile_cont</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.percentile_cont" title="datafusion.functions.percentile_cont"><code class="xref py py-obj docutils literal notranslate"><span class="pre">percentile_cont</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the exact percentile of input values using continuous interpolation.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.pi" title="datafusion.functions.pi"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pi</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.pi" title="datafusion.functions.pi"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pi</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an approximate value of π.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.position" title="datafusion.functions.position"><code class="xref py py-obj docutils literal notranslate"><span class="pre">position</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.position" title="datafusion.functions.position"><code class="xref py py-obj docutils literal notranslate"><span class="pre">position</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Finds the position from where the <code class="docutils literal notranslate"><span class="pre">substring</span></code> matches the <code class="docutils literal notranslate"><span class="pre">string</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.pow" title="datafusion.functions.pow"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pow</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.pow" title="datafusion.functions.pow"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pow</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns <code class="docutils literal notranslate"><span class="pre">base</span></code> raised to the power of <code class="docutils literal notranslate"><span class="pre">exponent</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.power" title="datafusion.functions.power"><code class="xref py py-obj docutils literal notranslate"><span class="pre">power</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.power" title="datafusion.functions.power"><code class="xref py py-obj docutils literal notranslate"><span class="pre">power</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns <code class="docutils literal notranslate"><span class="pre">base</span></code> raised to the power of <code class="docutils literal notranslate"><span class="pre">exponent</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.quantile_cont" title="datafusion.functions.quantile_cont"><code class="xref py py-obj docutils literal notranslate"><span class="pre">quantile_cont</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.quantile_cont" title="datafusion.functions.quantile_cont"><code class="xref py py-obj docutils literal notranslate"><span class="pre">quantile_cont</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the exact percentile of input values using continuous interpolation.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.radians" title="datafusion.functions.radians"><code class="xref py py-obj docutils literal notranslate"><span class="pre">radians</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.radians" title="datafusion.functions.radians"><code class="xref py py-obj docutils literal notranslate"><span class="pre">radians</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts the argument from degrees to radians.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.rand" title="datafusion.functions.rand"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rand</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns a random value in the range <code class="docutils literal notranslate"><span class="pre">0.0</span> <span class="pre"><=</span> <span class="pre">x</span> <span class="pre"><</span> <span class="pre">1.0</span></code>.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.random" title="datafusion.functions.random"><code class="xref py py-obj docutils literal notranslate"><span class="pre">random</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a random value in the range <code class="docutils literal notranslate"><span class="pre">0.0</span> <span class="pre"><=</span> <span class="pre">x</span> <span class="pre"><</span> <span class="pre">1.0</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.range" title="datafusion.functions.range"><code class="xref py py-obj docutils literal notranslate"><span class="pre">range</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Create a list of values in the range between start and stop.</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.range" title="datafusion.functions.range"><code class="xref py py-obj docutils literal notranslate"><span class="pre">range</span></code></a>(…)</p></td>
+<td><p>Create a list of values from <code class="docutils literal notranslate"><span class="pre">start</span></code> up to, but excluding, <code class="docutils literal notranslate"><span class="pre">stop</span></code>.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.rank" title="datafusion.functions.rank"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rank</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Create a rank window function.</p></td>
@@ -1373,7 +1427,7 @@ for categorized catalogs of aggregate and window functions.</p>
<td><p>Add right padding to a string.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.rtrim" title="datafusion.functions.rtrim"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rtrim</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Removes all characters, spaces by default, from the end of a string.</p></td>
+<td><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from the end of a string.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.sha224" title="datafusion.functions.sha224"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sha224</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the SHA-224 hash of a binary string.</p></td>
@@ -1430,7 +1484,7 @@ for categorized catalogs of aggregate and window functions.</p>
<td><p>Returns a struct with the given arguments.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.substr" title="datafusion.functions.substr"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substr</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Substring from the <code class="docutils literal notranslate"><span class="pre">position</span></code> to the end.</p></td>
+<td><p>Substring from the <code class="docutils literal notranslate"><span class="pre">position</span></code>, to the end or for <code class="docutils literal notranslate"><span class="pre">length</span></code> characters.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.substr_index" title="datafusion.functions.substr_index"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substr_index</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns an indexed substring.</p></td>
@@ -1438,94 +1492,97 @@ for categorized catalogs of aggregate and window functions.</p>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.substring" title="datafusion.functions.substring"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substring</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Substring from the <code class="docutils literal notranslate"><span class="pre">position</span></code> with <code class="docutils literal notranslate"><span class="pre">length</span></code> characters.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.sum" title="datafusion.functions.sum"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sum</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.substring_index" title="datafusion.functions.substring_index"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substring_index</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Returns an indexed substring.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.sum" title="datafusion.functions.sum"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sum</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the sum of a set of numbers.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.tan" title="datafusion.functions.tan"><code class="xref py py-obj docutils literal notranslate"><span class="pre">tan</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.tan" title="datafusion.functions.tan"><code class="xref py py-obj docutils literal notranslate"><span class="pre">tan</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the tangent of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.tanh" title="datafusion.functions.tanh"><code class="xref py py-obj docutils literal notranslate"><span class="pre">tanh</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.tanh" title="datafusion.functions.tanh"><code class="xref py py-obj docutils literal notranslate"><span class="pre">tanh</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the hyperbolic tangent of the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_char" title="datafusion.functions.to_char"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_char</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_char" title="datafusion.functions.to_char"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_char</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns a string representation of a date, time, timestamp or duration.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_date" title="datafusion.functions.to_date"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_date</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_date" title="datafusion.functions.to_date"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_date</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a value to a date (YYYY-MM-DD).</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_hex" title="datafusion.functions.to_hex"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_hex</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_hex" title="datafusion.functions.to_hex"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_hex</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts an integer to a hexadecimal string.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_local_time" title="datafusion.functions.to_local_time"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_local_time</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_local_time" title="datafusion.functions.to_local_time"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_local_time</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a timestamp with a timezone to a timestamp without a timezone.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_time" title="datafusion.functions.to_time"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_time</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_time" title="datafusion.functions.to_time"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_time</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a value to a time. Supports strings and timestamps as input.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp" title="datafusion.functions.to_timestamp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp" title="datafusion.functions.to_timestamp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string and optional formats to a <code class="docutils literal notranslate"><span class="pre">Timestamp</span></code> in nanoseconds.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_micros" title="datafusion.functions.to_timestamp_micros"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_micros</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_micros" title="datafusion.functions.to_timestamp_micros"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_micros</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string and optional formats to a <code class="docutils literal notranslate"><span class="pre">Timestamp</span></code> in microseconds.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_millis" title="datafusion.functions.to_timestamp_millis"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_millis</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_millis" title="datafusion.functions.to_timestamp_millis"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_millis</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string and optional formats to a <code class="docutils literal notranslate"><span class="pre">Timestamp</span></code> in milliseconds.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_nanos" title="datafusion.functions.to_timestamp_nanos"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_nanos</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_nanos" title="datafusion.functions.to_timestamp_nanos"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_nanos</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string and optional formats to a <code class="docutils literal notranslate"><span class="pre">Timestamp</span></code> in nanoseconds.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_seconds" title="datafusion.functions.to_timestamp_seconds"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_seconds</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_timestamp_seconds" title="datafusion.functions.to_timestamp_seconds"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_timestamp_seconds</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string and optional formats to a <code class="docutils literal notranslate"><span class="pre">Timestamp</span></code> in seconds.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.to_unixtime" title="datafusion.functions.to_unixtime"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_unixtime</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.to_unixtime" title="datafusion.functions.to_unixtime"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_unixtime</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string and optional formats to a Unixtime.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.translate" title="datafusion.functions.translate"><code class="xref py py-obj docutils literal notranslate"><span class="pre">translate</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.translate" title="datafusion.functions.translate"><code class="xref py py-obj docutils literal notranslate"><span class="pre">translate</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Replaces the characters in <code class="docutils literal notranslate"><span class="pre">from_val</span></code> with the counterpart in <code class="docutils literal notranslate"><span class="pre">to_val</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.trim" title="datafusion.functions.trim"><code class="xref py py-obj docutils literal notranslate"><span class="pre">trim</span></code></a>(→ datafusion.expr.Expr)</p></td>
-<td><p>Removes all characters, spaces by default, from both sides of a string.</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.trim" title="datafusion.functions.trim"><code class="xref py py-obj docutils literal notranslate"><span class="pre">trim</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from both sides of a string.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.trunc" title="datafusion.functions.trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.trunc" title="datafusion.functions.trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Truncate the number toward zero with optional precision.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.try_cast_to_type" title="datafusion.functions.try_cast_to_type"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_cast_to_type</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.try_cast_to_type" title="datafusion.functions.try_cast_to_type"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_cast_to_type</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Casts <code class="docutils literal notranslate"><span class="pre">value</span></code> to the data type of <code class="docutils literal notranslate"><span class="pre">type_ref</span></code>, NULL on failure.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.union_extract" title="datafusion.functions.union_extract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">union_extract</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.union_extract" title="datafusion.functions.union_extract"><code class="xref py py-obj docutils literal notranslate"><span class="pre">union_extract</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Extracts a value from a union type by field name.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.union_tag" title="datafusion.functions.union_tag"><code class="xref py py-obj docutils literal notranslate"><span class="pre">union_tag</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.union_tag" title="datafusion.functions.union_tag"><code class="xref py py-obj docutils literal notranslate"><span class="pre">union_tag</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the tag (active field name) of a union type.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.upper" title="datafusion.functions.upper"><code class="xref py py-obj docutils literal notranslate"><span class="pre">upper</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.upper" title="datafusion.functions.upper"><code class="xref py py-obj docutils literal notranslate"><span class="pre">upper</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Converts a string to uppercase.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.uuid" title="datafusion.functions.uuid"><code class="xref py py-obj docutils literal notranslate"><span class="pre">uuid</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.uuid" title="datafusion.functions.uuid"><code class="xref py py-obj docutils literal notranslate"><span class="pre">uuid</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns uuid v4 as a string value.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.var" title="datafusion.functions.var"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.var" title="datafusion.functions.var"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the sample variance of the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.var_pop" title="datafusion.functions.var_pop"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_pop</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.var_pop" title="datafusion.functions.var_pop"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_pop</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the population variance of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.var_population" title="datafusion.functions.var_population"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_population</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.var_population" title="datafusion.functions.var_population"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_population</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the population variance of the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.var_samp" title="datafusion.functions.var_samp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_samp</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.var_samp" title="datafusion.functions.var_samp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_samp</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the sample variance of the argument.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.var_sample" title="datafusion.functions.var_sample"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_sample</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.var_sample" title="datafusion.functions.var_sample"><code class="xref py py-obj docutils literal notranslate"><span class="pre">var_sample</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Computes the sample variance of the argument.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.version" title="datafusion.functions.version"><code class="xref py py-obj docutils literal notranslate"><span class="pre">version</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.version" title="datafusion.functions.version"><code class="xref py py-obj docutils literal notranslate"><span class="pre">version</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Returns the DataFusion version string.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.when" title="datafusion.functions.when"><code class="xref py py-obj docutils literal notranslate"><span class="pre">when</span></code></a>(→ datafusion.expr.CaseBuilder)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.when" title="datafusion.functions.when"><code class="xref py py-obj docutils literal notranslate"><span class="pre">when</span></code></a>(→ datafusion.expr.CaseBuilder)</p></td>
<td><p>Create a case expression that has no base expression.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.with_metadata" title="datafusion.functions.with_metadata"><code class="xref py py-obj docutils literal notranslate"><span class="pre">with_metadata</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.with_metadata" title="datafusion.functions.with_metadata"><code class="xref py py-obj docutils literal notranslate"><span class="pre">with_metadata</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Attaches Arrow field metadata (key/value pairs) to the input expression.</p></td>
</tr>
</tbody>
@@ -1624,6 +1681,45 @@ for categorized catalogs of aggregate and window functions.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.any_value">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">any_value</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
+<dd><p>Returns an arbitrary non-null value from each group.</p>
+<p>Returns NULL if every value in the group is NULL. Which value is returned
+is not specified and may differ between runs.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>expression</strong> – Argument to pick a value from</p></li>
+<li><p><strong>filter</strong> – If provided, only consider rows for which the filter is True</p></li>
+</ul>
+</dd>
+</dl>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="kc">None</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="kc">None</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">(</span>
+<span class="gp">... </span> <span class="p">[],</span> <span class="p">[</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">any_value</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span cla [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">7</span>
+</pre></div>
+</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="kc">None</span><span class="p">,</span> <span class="mi">7</span><span class="p">,</span> <span class="m [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">(</span>
+<span class="gp">... </span> <span class="p">[],</span> <span class="p">[</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">any_value</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span>
+<span class="gp">... </span> <span class="nb">filter</span><span class="o">=</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">)</span> <span class="o">></span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)]</span>
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">8</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.approx_distinct">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">approx_distinct</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class=" [...]
@@ -1631,7 +1727,7 @@ for categorized catalogs of aggregate and window functions.</p>
<p>This aggregate function is similar to <a class="reference internal" href="#datafusion.functions.count" title="datafusion.functions.count"><code class="xref py py-func docutils literal notranslate"><span class="pre">count()</span></code></a> with distinct set, but it
will approximate the number of distinct entries. It may return significantly faster
than <a class="reference internal" href="#datafusion.functions.count" title="datafusion.functions.count"><code class="xref py py-func docutils literal notranslate"><span class="pre">count()</span></code></a> for some DataFrames.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -1669,7 +1765,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Returns the approximate median value.</p>
<p>This aggregate function is similar to <a class="reference internal" href="#datafusion.functions.median" title="datafusion.functions.median"><code class="xref py py-func docutils literal notranslate"><span class="pre">median()</span></code></a>, but it will only
approximate the median. It may return significantly faster for some DataFrames.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -1713,8 +1809,10 @@ between two of the values.</p>
<p>This function uses the [t-digest](<a class="reference external" href="https://arxiv.org/abs/1902.04023">https://arxiv.org/abs/1902.04023</a>) algorithm to
compute the percentile. You can limit the number of bins used in this algorithm by
setting the <code class="docutils literal notranslate"><span class="pre">num_centroids</span></code> parameter.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the options <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code> and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>. A chained <code class="docutils literal notranslate"><span class="pre">order_by</span></code> sets only
+the sort direction; the percentile is still computed over <code class="docutils literal notranslate"><span class="pre">sort_expression</span></code>,
+so pass the same expression.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
@@ -1754,8 +1852,10 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Returns the value of the weighted approximate percentile.</p>
<p>This aggregate function is similar to <a class="reference internal" href="#datafusion.functions.approx_percentile_cont" title="datafusion.functions.approx_percentile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">approx_percentile_cont()</span></code></a> except that
it uses the associated associated weights.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the option <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code> and rejects <code class="docutils literal notranslate"><span class="pre">distinct</span></code>. A chained <code class="docutils literal notranslate"><span class="pre">order_by</span></code>
+sets only the sort direction; the percentile is still computed over
+<code class="docutils literal notranslate"><span class="pre">sort_expression</span></code>, so pass the same expression.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
@@ -1800,6 +1900,26 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_add">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_add</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
+<dd><p>Returns the element-wise sum of two numeric arrays of equal length.</p>
+<p>A NULL element in either input produces a NULL at that position. Execution
+fails if the arrays in a row have different lengths.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">(</span>
+<span class="gp">... </span> <span class="p">{</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]],</span> <span class="s2">"b"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">10.0</span><span class="p">,</span> <span class="mf">20.0</span><span class="p">,</span [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_add</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</spa [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">[11.0, 22.0, 33.0]</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_agg">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_agg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
@@ -1807,7 +1927,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<p>Currently <code class="docutils literal notranslate"><span class="pre">distinct</span></code> and <code class="docutils literal notranslate"><span class="pre">order_by</span></code> cannot be used together. As a work around,
consider <a class="reference internal" href="#datafusion.functions.array_sort" title="datafusion.functions.array_sort"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_sort()</span></code></a> after aggregation.
[Issue Tracker](<a class="github reference external" href="https://github.com/apache/datafusion/issues/12371">apache/datafusion#12371</a>)</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the option <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -1912,6 +2032,24 @@ return a boolean expression.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_avg">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_avg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
+<dd><p>Returns the arithmetic mean of the elements of a numeric array.</p>
+<p>NULL elements are skipped and excluded from the count. Returns NULL if the
+array is NULL, empty, or contains only NULL elements.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">1.0</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]]})</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_avg</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)</span>
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">2.0</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_cat">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_cat</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title [...]
@@ -2100,6 +2238,37 @@ matching elements.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_first">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_first</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
+<dd><p>Return the first element of <code class="docutils literal notranslate"><span class="pre">array</span></code> for which <code class="docutils literal notranslate"><span class="pre">predicate</span></code> is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
+<p><code class="docutils literal notranslate"><span class="pre">predicate</span></code> may be a Python callable, converted to a lambda
+automatically, or an explicit lambda built with <a class="reference internal" href="#datafusion.functions.lambda_" title="datafusion.functions.lambda_"><code class="xref py py-func docutils literal notranslate"><span class="pre">lambda_()</span></code></a>. It must
+return a boolean expression. Returns NULL if no element matches.</p>
+<p class="rubric">Examples</p>
+<p>Using a Python callable:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">]]})</span>
+<span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">F</span><span class="o">.</span><span class="n">array_first</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="k">lambda</span> <span class="n">v</span><span class="p">:</span> <span class="n">v</span> <span class="o">></span> <span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span cl [...]
+<span class="gp">... </span><span class="p">)</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"f"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">3</span>
+</pre></div>
+</div>
+<p>Using an explicit lambda built with <a class="reference internal" href="#datafusion.functions.lambda_" title="datafusion.functions.lambda_"><code class="xref py py-func docutils literal notranslate"><span class="pre">lambda_()</span></code></a>:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">predicate</span> <span class="o">=</span> <span class="n">F</span><span class="o">.</span><span class="n">lambda_</span><span class="p">([</span><span class="s2">"v"</span><span class="p">],</span> <span class="n">F</span><span class="o">.</span><span class="n">lambda_var</span><span class="p">(</span><span class="s2">"v"</sp [...]
+<span class="gp">>>> </span><span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">F</span><span class="o">.</span><span class="n">array_first</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">predicate</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"f"</span><span class="p">)</span>
+<span class="gp">... </span><span class="p">)</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"f"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">3</span>
+</pre></div>
+</div>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p><a class="reference internal" href="#datafusion.functions.array_filter" title="datafusion.functions.array_filter"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_filter()</span></code></a>, <a class="reference internal" href="#datafusion.functions.array_any_match" title="datafusion.functions.array_any_match"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_any_match()</span></code></a>, <a class="reference internal" [...]
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_has">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_has</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">first_array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
@@ -2181,7 +2350,7 @@ array. Otherwise, it returns false.</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_join">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
<dd><p>Converts each element to its text representation.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
@@ -2369,6 +2538,24 @@ if any element of the input array is NULL.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_product">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_product</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<dd><p>Returns the product of the elements of a numeric array.</p>
+<p>NULL elements are skipped. Returns NULL if the array is NULL, empty, or
+contains only NULL elements.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">2.0</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]]})</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_product</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"result"</span><span class="p [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">6.0</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_push_back">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_push_back</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
@@ -2536,6 +2723,24 @@ be filled with the given <code class="docutils literal notranslate"><span class=
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_scale">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_scale</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
+<dd><p>Multiplies each element of a numeric array by <code class="docutils literal notranslate"><span class="pre">scalar</span></code>.</p>
+<p>A NULL element produces a NULL at that position. Returns NULL if <code class="docutils literal notranslate"><span class="pre">scalar</span></code>
+is NULL.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]]})</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_scale</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="mf">2.0</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">[2.0, 4.0, 6.0]</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_slice">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_slice</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
@@ -2595,17 +2800,68 @@ be filled with the given <code class="docutils literal notranslate"><span class=
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_subtract">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_subtract</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
+<dd><p>Returns the element-wise difference of two numeric arrays of equal length.</p>
+<p>Computes <code class="docutils literal notranslate"><span class="pre">array1[i]</span> <span class="pre">-</span> <span class="pre">array2[i]</span></code>. A NULL element in either input produces
+a NULL at that position. Execution fails if the arrays in a row have
+different lengths.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">(</span>
+<span class="gp">... </span> <span class="p">{</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">10.0</span><span class="p">,</span> <span class="mf">20.0</span><span class="p">,</span> <span class="mf">30.0</span><span class="p">]],</span> <span class="s2">"b"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">2.0</span><span class="p">,</spa [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_subtract</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">)</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)</span>
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">[9.0, 18.0, 27.0]</span>
+</pre></div>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.array_sum">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_sum</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
+<dd><p>Returns the sum of the elements of a numeric array.</p>
+<p>NULL elements are skipped. Returns NULL if the array is NULL, empty, or
+contains only NULL elements.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mf">1.0</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">]]})</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_sum</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">))</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)</span>
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">4.0</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.array_to_string">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_to_string</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">array_to_string</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
<dd><p>Converts each element to its text representation.</p>
+<p>NULL elements are omitted unless <code class="docutils literal notranslate"><span class="pre">null_string</span></code> is given, in which case it
+is written in their place.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
-<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]]})</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mi">1</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="mi">3</span><span class="p">]]})</span>
<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_to_string</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="s2">","</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p"> [...]
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"s"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
-<span class="go">'1,2,3'</span>
+<span class="go">'1,3'</span>
+</pre></div>
+</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">array_to_string</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="s2">","</span><span class="p">,</span> <span class="n">null_string</span><span class="o">=</span><span class="s2">"*"</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"s"</span><span class="p">))</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"s"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'1,*,3'</span>
</pre></div>
</div>
</dd></dl>
@@ -2903,7 +3159,7 @@ type syntax (for example <code class="docutils literal notranslate"><span class=
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">avg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
<dd><p>Returns the average value.</p>
<p>This aggregate function expects a numeric expression and will return a float.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -2947,15 +3203,16 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.bit_and">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">bit_and</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">bit_and</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
<dd><p>Computes the bitwise AND of the argument.</p>
<p>This aggregate function will bitwise compare every value in the input partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>expression</strong> – Argument to perform bitwise calculation on</p></li>
+<li><p><strong>distinct</strong> – If True, evaluate each unique value of expression only once</p></li>
<li><p><strong>filter</strong> – If provided, only compute against rows for which the filter is True</p></li>
</ul>
</dd>
@@ -2999,15 +3256,16 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.bit_or">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">bit_or</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">bit_or</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
<dd><p>Computes the bitwise OR of the argument.</p>
<p>This aggregate function will bitwise compare every value in the input partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>expression</strong> – Argument to perform bitwise calculation on</p></li>
+<li><p><strong>distinct</strong> – If True, evaluate each unique value of expression only once</p></li>
<li><p><strong>filter</strong> – If provided, only compute against rows for which the filter is True</p></li>
</ul>
</dd>
@@ -3042,7 +3300,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">bit_xor</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
<dd><p>Computes the bitwise XOR of the argument.</p>
<p>This aggregate function will bitwise compare every value in the input partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -3084,7 +3342,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the boolean AND of the argument.</p>
<p>This aggregate function will compare every value in the input partition. These are
expected to be boolean values.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -3126,7 +3384,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the boolean OR of the argument.</p>
<p>This aggregate function will compare every value in the input partition. These are
expected to be boolean values.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -3164,8 +3422,8 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.btrim">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">btrim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.e [...]
-<dd><p>Removes all characters, spaces by default, from both sides of a string.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">btrim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.e [...]
+<dd><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from both sides of a string.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">" a "</span><span class="p">]})</span>
@@ -3174,6 +3432,15 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<span class="go">'a'</span>
</pre></div>
</div>
+<p>Trim a different set of characters:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"xxaxx"</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">trim_df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">btrim</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">characters</span><span class="o">=</span><span class="s2">"x"</span><span class="p">)</span><span class="o">.</sp [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">trim_df</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"trimmed"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'a'</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -3391,7 +3658,7 @@ pass a type string or <code class="docutils literal notranslate"><span class="pr
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">corr</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value_y</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusio [...]
<dd><p>Returns the correlation coefficient between <code class="docutils literal notranslate"><span class="pre">value1</span></code> and <code class="docutils literal notranslate"><span class="pre">value2</span></code>.</p>
<p>This aggregate function expects both values to be numeric and will return a float.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -3525,7 +3792,7 @@ function returns NULL.</p>
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">count</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expressions</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
<dd><p>Returns the number of rows that match the given arguments.</p>
<p>This aggregate function will count the non-null rows provided in the expression.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -3564,8 +3831,9 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">count_star</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">filter</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
<dd><p>Create a COUNT(1) aggregate expression.</p>
<p>This aggregate function will count all of the rows in the partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">distinct</span></code>, and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>. <code class="docutils literal notranslate"><span class="pre">distinct</span></code> counts the distinct
+values of the constant <code class="docutils literal notranslate"><span class="pre">1</span></code>, so the result is 1 for any non-empty input.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>filter</strong> – If provided, only count rows for which the filter is True</p>
@@ -3606,7 +3874,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">covar_pop</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value_y</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
<dd><p>Computes the population covariance.</p>
<p>This aggregate function expects both values to be numeric and will return a float.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -3650,7 +3918,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">covar_samp</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value_y</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
<dd><p>Computes the sample covariance.</p>
<p>This aggregate function expects both values to be numeric and will return a float.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -4094,6 +4362,32 @@ blake2b, and blake3.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.file_row_index">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">file_row_index</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span> [...]
+<dd><p>Returns the zero-based position of the current row within its source file.</p>
+<p>The index restarts at zero for each file, so rows from different files in one
+scan can share a value. Only valid inside a scan of a Parquet table;
+evaluating it anywhere else raises an error.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">tempfile</span><span class="o">,</span><span class="w"> </span><span class="nn">os</span>
+<span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span><span class="o">,</span><span class="w"> </span><span class="nn">pyarrow.parquet</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pq</span>
+<span class="gp">>>> </span><span class="n">tmp</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">mkdtemp</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp</span><span class="p">,</span> <span class="s2">"data.parquet"</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">pq</span><span class="o">.</span><span class="n">write_table</span><span class="p">(</span><span class="n">pa</span><span class="o">.</span><span class="n">table</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="mi">10</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">30</span><span class="p">]}),</span> <spa [...]
+<span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">read_parquet</span><span class="p">(</span><span class="n">path</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">file_row_index</span><span class="p">()</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"i"</span><span class="p">) [...]
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"i"</span><span class="p">)</span><span class="o">.</span><span class="n">to_pylist</span><span class="p">()</span>
+<span class="go">[1, 2]</span>
+</pre></div>
+</div>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p><a class="reference internal" href="#datafusion.functions.input_file_name" title="datafusion.functions.input_file_name"><code class="xref py py-func docutils literal notranslate"><span class="pre">input_file_name()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.find_in_set">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">find_in_set</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
@@ -4117,7 +4411,7 @@ blake2b, and blake3.</p>
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">first_value</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
<dd><p>Returns the first value in a group of values.</p>
<p>This aggregate function will return the first value in the partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the option <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -4221,25 +4515,29 @@ column names or expressions.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.gen_series">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">gen_series</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
-<dd><p>Creates a list of values in the range between start and stop.</p>
-<p>Unlike <a class="reference internal" href="#datafusion.functions.range" title="datafusion.functions.range"><code class="xref py py-func docutils literal notranslate"><span class="pre">range()</span></code></a>, this includes the upper bound.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">gen_series</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">stop</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
+<dt class="sig sig-object py">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">gen_series</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
+<dd><p>Creates a list of values from <code class="docutils literal notranslate"><span class="pre">start</span></code> up to and including <code class="docutils literal notranslate"><span class="pre">stop</span></code>.</p>
+<p>Unlike <a class="reference internal" href="#datafusion.functions.range" title="datafusion.functions.range"><code class="xref py py-func docutils literal notranslate"><span class="pre">range()</span></code></a>, this includes the upper bound. With a single
+argument it is <code class="docutils literal notranslate"><span class="pre">stop</span></code> and the series starts at 0.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="mi">0</span><span class="p">]})</span>
-<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
-<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">gen_series</span><span class="p">(</span>
-<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span>
-<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"result"</span><span class="p">))</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">gen_series</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span clas [...]
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">[0, 1, 2, 3]</span>
+</pre></div>
+</div>
+<p>Specify a <code class="docutils literal notranslate"><span class="pre">stop</span></code>:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">gen_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,< [...]
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
<span class="go">[1, 2, 3, 4, 5]</span>
</pre></div>
</div>
-<p>Specify a custom <code class="docutils literal notranslate"><span class="pre">step</span></code>:</p>
+<p>Specify a <code class="docutils literal notranslate"><span class="pre">step</span></code>:</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
-<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">gen_series</span><span class="p">(</span>
-<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">10</span><span class="p">),</span> <span class="n">step</span><span class="o">=</span><span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><s [...]
-<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"result"</span><span class="p">))</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">gen_series</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">stop</span><span class="o">=</span><span class="mi">10</span><span class="p">,</span> <span class="n">step</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span><span class="o">.</span><span class="n">alias [...]
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"result"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
<span class="go">[1, 4, 7, 10]</span>
</pre></div>
@@ -4248,9 +4546,10 @@ column names or expressions.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.generate_series">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">generate_series</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
-<dd><p>Creates a list of values in the range between start and stop.</p>
-<p>Unlike <a class="reference internal" href="#datafusion.functions.range" title="datafusion.functions.range"><code class="xref py py-func docutils literal notranslate"><span class="pre">range()</span></code></a>, this includes the upper bound.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">generate_series</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">stop</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
+<dt class="sig sig-object py">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">generate_series</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
+<dd><p>Creates a list of values from <code class="docutils literal notranslate"><span class="pre">start</span></code> up to and including <code class="docutils literal notranslate"><span class="pre">stop</span></code>.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
<p>This is an alias for <a class="reference internal" href="#datafusion.functions.gen_series" title="datafusion.functions.gen_series"><code class="xref py py-func docutils literal notranslate"><span class="pre">gen_series()</span></code></a>.</p>
@@ -4491,6 +4790,31 @@ either array is NULL.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.input_file_name">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">input_file_name</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span [...]
+<dd><p>Returns the path of the file that produced the current row.</p>
+<p>Only valid inside a scan of a file-backed table; evaluating it anywhere
+else raises an error.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">tempfile</span><span class="o">,</span><span class="w"> </span><span class="nn">os</span>
+<span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span><span class="o">,</span><span class="w"> </span><span class="nn">pyarrow.parquet</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pq</span>
+<span class="gp">>>> </span><span class="n">tmp</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">mkdtemp</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">tmp</span><span class="p">,</span> <span class="s2">"data.parquet"</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">pq</span><span class="o">.</span><span class="n">write_table</span><span class="p">(</span><span class="n">pa</span><span class="o">.</span><span class="n">table</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">]}),</span> <span class="n">path</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">read_parquet</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">input_file_name</span><span class="p">()</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"f"</span><span class="p"> [...]
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"f"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">"data.parquet"</span><span class="p">)</span>
+<span class="go">True</span>
+</pre></div>
+</div>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p><a class="reference internal" href="#datafusion.functions.file_row_index" title="datafusion.functions.file_row_index"><code class="xref py py-func docutils literal notranslate"><span class="pre">file_row_index()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.instr">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">instr</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusio [...]
@@ -4537,7 +4861,7 @@ either array is NULL.</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.lag">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">lag</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.exp [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">lag</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.exp [...]
<dd><p>Create a lag window function.</p>
<p>Lag operation will return the argument that is in the previous shift_offset-th row
in the partition. For example <code class="docutils literal notranslate"><span class="pre">lag(col("b"),</span> <span class="pre">shift_offset=3,</span> <span class="pre">default_value=5)</span></code>
@@ -4564,6 +4888,8 @@ functions on a simple DataFrame:</p>
<li><p><strong>partition_by</strong> – Expressions to partition the window frame on.</p></li>
<li><p><strong>order_by</strong> – Set ordering within the window frame. Accepts
column names or expressions.</p></li>
+<li><p><strong>null_treatment</strong> – Set to <code class="docutils literal notranslate"><span class="pre">IGNORE_NULLS</span></code> to skip null values when
+counting <code class="docutils literal notranslate"><span class="pre">shift_offset</span></code> rows.</p></li>
</ul>
</dd>
</dl>
@@ -4591,6 +4917,17 @@ column names or expressions.</p></li>
<span class="go">[0, 1, 0]</span>
</pre></div>
</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"i"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi"> [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"i"</span><span class="p">),</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">lag</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="n">order_by</span><span class="o">=</span><span class="s2">"i"</span><span class="p">,</span>
+<span class="gp">... </span> <span class="n">null_treatment</span><span class="o">=</span><span class="n">dfn</span><span class="o">.</span><span class="n">common</span><span class="o">.</span><span class="n">NullTreatment</span><span class="o">.</span><span class="n">IGNORE_NULLS</span><span class="p">,</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"lag"</span><span class="p">))</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"i"</span><span class="p">))</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"lag"</span><span class="p">)</span><span class="o">.</span><span class="n">t [...]
+<span class="go">[None, 1, 1, 1]</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -4655,7 +4992,7 @@ during query planning.</p>
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">last_value</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
<dd><p>Returns the last value in a group of values.</p>
<p>This aggregate function will return the last value in the partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the option <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -4713,7 +5050,7 @@ column names or expressions.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.lead">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">lead</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.ex [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">lead</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.ex [...]
<dd><p>Create a lead window function.</p>
<p>Lead operation will return the argument that is in the next shift_offset-th row in
the partition. For example <code class="docutils literal notranslate"><span class="pre">lead(col("b"),</span> <span class="pre">shift_offset=3,</span> <span class="pre">default_value=5)</span></code> will
@@ -4732,7 +5069,7 @@ functions on a simple DataFrame:</p>
</pre></div>
</div>
<p>To set window function parameters use the window builder approach described in the
-ref:<cite>_window_functions</cite> online documentation.</p>
+<a class="reference internal" href="../../../user-guide/common-operations/windows.html#window-functions"><span class="std std-ref">Window Functions</span></a> online documentation.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
@@ -4742,6 +5079,8 @@ ref:<cite>_window_functions</cite> online documentation.</p>
<li><p><strong>partition_by</strong> – Expressions to partition the window frame on.</p></li>
<li><p><strong>order_by</strong> – Set ordering within the window frame. Accepts
column names or expressions.</p></li>
+<li><p><strong>null_treatment</strong> – Set to <code class="docutils literal notranslate"><span class="pre">IGNORE_NULLS</span></code> to skip null values when
+counting <code class="docutils literal notranslate"><span class="pre">shift_offset</span></code> rows.</p></li>
</ul>
</dd>
</dl>
@@ -4769,6 +5108,17 @@ column names or expressions.</p></li>
<span class="go">[2, 0, 0]</span>
</pre></div>
</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"i"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi"> [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"i"</span><span class="p">),</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">lead</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="n">order_by</span><span class="o">=</span><span class="s2">"i"</span><span class="p">,</span>
+<span class="gp">... </span> <span class="n">null_treatment</span><span class="o">=</span><span class="n">dfn</span><span class="o">.</span><span class="n">common</span><span class="o">.</span><span class="n">NullTreatment</span><span class="o">.</span><span class="n">IGNORE_NULLS</span><span class="p">,</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"lead"</span><span class="p">))</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"i"</span><span class="p">))</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"lead"</span><span class="p">)</span><span class="o">.</span><span class="n"> [...]
+<span class="go">[4, 4, 4, None]</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -4833,6 +5183,16 @@ column names or expressions.</p></li>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_add">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_add</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
+<dd><p>Returns the element-wise sum of two numeric lists of equal length.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_add" title="datafusion.functions.array_add"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_add()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_any_match">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_any_match</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
@@ -4863,6 +5223,16 @@ column names or expressions.</p></li>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_avg">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_avg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
+<dd><p>Returns the arithmetic mean of the elements of a numeric list.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_avg" title="datafusion.functions.array_avg"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_avg()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_cat">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_cat</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title= [...]
@@ -4983,6 +5353,16 @@ column names or expressions.</p></li>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_first">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_first</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
+<dd><p>Return the first element of a list for which a predicate is <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_first" title="datafusion.functions.array_first"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_first()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_has">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_has</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
@@ -5035,7 +5415,7 @@ column names or expressions.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_join">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
<dd><p>Converts each element to its text representation.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
@@ -5153,6 +5533,16 @@ column names or expressions.</p></li>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_product">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_product</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
+<dd><p>Returns the product of the elements of a numeric list.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_product" title="datafusion.functions.array_product"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_product()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_push_back">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_push_back</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
@@ -5267,6 +5657,16 @@ filled with the given <code class="docutils literal notranslate"><span class="pr
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_scale">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_scale</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
+<dd><p>Multiplies each element of a numeric list by a scalar.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_scale" title="datafusion.functions.array_scale"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_scale()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_slice">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_slice</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
@@ -5287,9 +5687,29 @@ filled with the given <code class="docutils literal notranslate"><span class="pr
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_subtract">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_subtract</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
+<dd><p>Returns the element-wise difference of two numeric lists of equal length.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_subtract" title="datafusion.functions.array_subtract"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_subtract()</span></code></a>.</p>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.list_sum">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_sum</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">array</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
+<dd><p>Returns the sum of the elements of a numeric list.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.array_sum" title="datafusion.functions.array_sum"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_sum()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.list_to_string">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_to_string</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">list_to_string</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
<dd><p>Converts each element to its text representation.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
@@ -5430,8 +5850,8 @@ truncated (on the right).</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.ltrim">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">ltrim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.e [...]
-<dd><p>Removes all characters, spaces by default, from the beginning of a string.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">ltrim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.e [...]
+<dd><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from the beginning of a string.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">" a "</span><span class="p">]})</span>
@@ -5440,6 +5860,15 @@ truncated (on the right).</p>
<span class="go">'a '</span>
</pre></div>
</div>
+<p>Trim a different set of characters:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"xxaxx"</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">trim_df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">ltrim</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">characters</span><span class="o">=</span><span class="s2">"x"</span><span class="p">)</span><span class="o">.</sp [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">trim_df</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"trimmed"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'axx'</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -5637,7 +6066,7 @@ are automatically converted to literal expressions.</p>
<dt class="sig sig-object py" id="datafusion.functions.max">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">max</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
<dd><p>Aggregate function that returns the maximum value of the argument.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -5685,7 +6114,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.mean">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">mean</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">mean</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafu [...]
<dd><p>Returns the average (mean) value of the argument.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
@@ -5699,7 +6128,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the median of a set of numbers.</p>
<p>This aggregate function returns the median value of the expression for the given
aggregate function.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -5737,7 +6166,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dt class="sig sig-object py" id="datafusion.functions.min">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">min</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
<dd><p>Aggregate function that returns the minimum value of the argument.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -5838,7 +6267,7 @@ require pandas to convert to Python datetime objects.</p>
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">nth_value</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
<dd><p>Returns the n-th value in a group of values.</p>
<p>This aggregate function will return the n-th value in the partition.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the option <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6107,17 +6536,20 @@ column names or expressions.</p></li>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.percentile_cont">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">percentile_cont</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">sort_expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span cl [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">percentile_cont</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">sort_expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span cl [...]
<dd><p>Computes the exact percentile of input values using continuous interpolation.</p>
<p>Unlike <a class="reference internal" href="#datafusion.functions.approx_percentile_cont" title="datafusion.functions.approx_percentile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">approx_percentile_cont()</span></code></a>, this function computes the exact
percentile value rather than an approximation.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the option <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>. A chained <code class="docutils literal notranslate"><span class="pre">order_by</span></code> sets only the sort
+direction; the percentile is still computed over <code class="docutils literal notranslate"><span class="pre">sort_expression</span></code>, so pass
+the same expression.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>sort_expression</strong> – Values for which to find the percentile</p></li>
<li><p><strong>percentile</strong> – This must be between 0.0 and 1.0, inclusive</p></li>
+<li><p><strong>distinct</strong> – If True, duplicate values are removed before computing</p></li>
<li><p><strong>filter</strong> – If provided, only compute against rows for which the filter is True</p></li>
</ul>
</dd>
@@ -6142,6 +6574,15 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<span class="go">3.5</span>
</pre></div>
</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class=" [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">(</span>
+<span class="gp">... </span> <span class="p">[],</span> <span class="p">[</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">percentile_cont</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="mf">0.5</span><span class="p">,</span> <span class="n">distinct</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)])</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">2.5</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -6199,7 +6640,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.quantile_cont">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">quantile_cont</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">sort_expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span clas [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">quantile_cont</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">sort_expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span clas [...]
<dd><p>Computes the exact percentile of input values using continuous interpolation.</p>
<div class="admonition seealso">
<p class="admonition-title">See also</p>
@@ -6224,6 +6665,16 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.rand">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">rand</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a></span [...]
+<dd><p>Returns a random value in the range <code class="docutils literal notranslate"><span class="pre">0.0</span> <span class="pre"><=</span> <span class="pre">x</span> <span class="pre"><</span> <span class="pre">1.0</span></code>.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.random" title="datafusion.functions.random"><code class="xref py py-func docutils literal notranslate"><span class="pre">random()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.random">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">random</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a></sp [...]
@@ -6243,13 +6694,28 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.range">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">range</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion [...]
-<dd><p>Create a list of values in the range between start and stop.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">range</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">stop</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion. [...]
+<dt class="sig sig-object py">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">range</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion [...]
+<dd><p>Create a list of values from <code class="docutils literal notranslate"><span class="pre">start</span></code> up to, but excluding, <code class="docutils literal notranslate"><span class="pre">stop</span></code>.</p>
+<p>With a single argument it is <code class="docutils literal notranslate"><span class="pre">stop</span></code> and the range starts at 0, like
+Python’s built-in <a class="reference internal" href="#datafusion.functions.range" title="datafusion.functions.range"><code class="xref py py-class docutils literal notranslate"><span class="pre">range</span></code></a>.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
-<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
-<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">range</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">0</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="mi">5</span><span class="p">),</span> [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">range</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2 [...]
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"r"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">[0, 1, 2, 3, 4]</span>
+</pre></div>
+</div>
+<p>Specify a <code class="docutils literal notranslate"><span class="pre">stop</span></code>:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span [...]
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"r"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">[1, 2, 3, 4]</span>
+</pre></div>
+</div>
+<p>Specify a <code class="docutils literal notranslate"><span class="pre">step</span></code>:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span [...]
<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"r"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
<span class="go">[0, 2, 4]</span>
</pre></div>
@@ -6485,7 +6951,7 @@ corresponding index in <code class="docutils literal notranslate"><span class="p
<dd><p>Computes the average of the independent variable <code class="docutils literal notranslate"><span class="pre">x</span></code>.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6524,7 +6990,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the average of the dependent variable <code class="docutils literal notranslate"><span class="pre">y</span></code>.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6563,7 +7029,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Counts the number of rows in which both expressions are not null.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6602,7 +7068,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the intercept from the linear regression.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6643,7 +7109,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the R-squared value from linear regression.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6682,7 +7148,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the slope from linear regression.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6721,7 +7187,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the sum of squares of the independent variable <code class="docutils literal notranslate"><span class="pre">x</span></code>.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6760,7 +7226,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the sum of products of pairs of numbers.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6799,7 +7265,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dd><p>Computes the sum of squares of the dependent variable <code class="docutils literal notranslate"><span class="pre">y</span></code>.</p>
<p>This is a linear regression aggregate function. Only non-null pairs of the inputs
are evaluated.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -6986,8 +7452,8 @@ by default). If the string is already longer than length then it is truncated.</
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.rtrim">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">rtrim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.e [...]
-<dd><p>Removes all characters, spaces by default, from the end of a string.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">rtrim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.e [...]
+<dd><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from the end of a string.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">" a "</span><span class="p">]})</span>
@@ -6996,6 +7462,15 @@ by default). If the string is already longer than length then it is truncated.</
<span class="go">' a'</span>
</pre></div>
</div>
+<p>Trim a different set of characters:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"xxaxx"</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">trim_df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">rtrim</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">characters</span><span class="o">=</span><span class="s2">"x"</span><span class="p">)</span><span class="o">.</sp [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">trim_df</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"trimmed"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'xxa'</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -7154,7 +7629,7 @@ on the index.</p>
<dt class="sig sig-object py" id="datafusion.functions.stddev">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">stddev</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
<dd><p>Computes the standard deviation of the argument.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -7190,7 +7665,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dt class="sig sig-object py" id="datafusion.functions.stddev_pop">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">stddev_pop</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
<dd><p>Computes the population standard deviation of the argument.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -7237,18 +7712,19 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.string_agg">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">string_agg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">string_agg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
<dd><p>Concatenates the input strings.</p>
<p>This aggregate function will concatenate input strings, ignoring null values, and
separating them with the specified delimiter. Non-string values will be converted to
their string equivalents.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
-the options <code class="docutils literal notranslate"><span class="pre">distinct</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
+the option <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
<li><p><strong>expression</strong> – Argument to perform bitwise calculation on</p></li>
<li><p><strong>delimiter</strong> – Text to place between each value of expression</p></li>
+<li><p><strong>distinct</strong> – If True, each unique value of expression is included only once</p></li>
<li><p><strong>filter</strong> – If provided, only compute against rows for which the filter is True</p></li>
<li><p><strong>order_by</strong> – Set the ordering of the expression to evaluate. Accepts
column names or expressions.</p></li>
@@ -7276,6 +7752,15 @@ column names or expressions.</p></li>
<span class="go">'y,z'</span>
</pre></div>
</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"y"</span><span class="p">,</span> <span class="s2">"x"</span><span class="p">,< [...]
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">aggregate</span><span class="p">(</span>
+<span class="gp">... </span> <span class="p">[],</span> <span class="p">[</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">string_agg</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="s2">","</span><span class="p">,</span> <span class="n">distinct</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">order_by</span><span class="o">=</span><span class="s2">"a"</span><span class="p">,</span>
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"s"</span><span class="p">)])</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"s"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'x,y'</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -7352,8 +7837,8 @@ for <cite>N</cite> children.</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.substr">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">substr</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusi [...]
-<dd><p>Substring from the <code class="docutils literal notranslate"><span class="pre">position</span></code> to the end.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">substr</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusi [...]
+<dd><p>Substring from the <code class="docutils literal notranslate"><span class="pre">position</span></code>, to the end or for <code class="docutils literal notranslate"><span class="pre">length</span></code> characters.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"hello"</span><span class="p">]})</span>
@@ -7363,6 +7848,16 @@ for <cite>N</cite> children.</p>
<span class="go">'llo'</span>
</pre></div>
</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">substr</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="mi">2</span><span class="p">,</span> <span class="n">length</span><span class="o">=</span><span class="mi">3</span><span clas [...]
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"s"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'ell'</span>
+</pre></div>
+</div>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p><a class="reference internal" href="#datafusion.functions.substring" title="datafusion.functions.substring"><code class="xref py py-func docutils literal notranslate"><span class="pre">substring()</span></code></a>.</p>
+</div>
</dd></dl>
<dl class="py function">
@@ -7397,12 +7892,22 @@ for <cite>N</cite> children.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.substring_index">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">substring_index</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">string</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
+<dd><p>Returns an indexed substring.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.substr_index" title="datafusion.functions.substr_index"><code class="xref py py-func docutils literal notranslate"><span class="pre">substr_index()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.sum">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">sum</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafus [...]
<dd><p>Computes the sum of a set of numbers.</p>
<p>This aggregate function expects a numeric expression.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code> and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -7763,8 +8268,8 @@ HH:MM:SS.nnnnnnnnn;</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.trim">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">trim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.ex [...]
-<dd><p>Removes all characters, spaces by default, from both sides of a string.</p>
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">trim</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">arg</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.ex [...]
+<dd><p>Removes <code class="docutils literal notranslate"><span class="pre">characters</span></code>, spaces by default, from both sides of a string.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">" hello "</span><span class="p">]})</span>
@@ -7773,6 +8278,15 @@ HH:MM:SS.nnnnnnnnn;</p>
<span class="go">'hello'</span>
</pre></div>
</div>
+<p>Trim a different set of characters:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"xxhelloxx"</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">result</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">trim</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">characters</span><span class="o">=</span><span class="s2">"x"</span><span class="p">)</span><span class="o">.</spa [...]
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">result</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"t"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'hello'</span>
+</pre></div>
+</div>
</dd></dl>
<dl class="py function">
@@ -7911,7 +8425,7 @@ variant, otherwise returns NULL.</p>
<dt class="sig sig-object py" id="datafusion.functions.var_pop">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">var_pop</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
<dd><p>Computes the population variance of the argument.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -7957,7 +8471,7 @@ the options <code class="docutils literal notranslate"><span class="pre">order_b
<dt class="sig sig-object py" id="datafusion.functions.var_samp">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.</span></span><span class="sig-name descname"><span class="pre">var_samp</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">expression</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
<dd><p>Computes the sample variance of the argument.</p>
-<p>If using the builder functions described in ref:<cite>_aggregation</cite> this function ignores
+<p>If using the builder functions described in <a class="reference internal" href="../../../user-guide/common-operations/aggregations.html#aggregation"><span class="std std-ref">Aggregation</span></a> this function ignores
the options <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -8121,15 +8635,18 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.acosh"><code class="docutils literal notranslate"><span class="pre">acosh()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.alias"><code class="docutils literal notranslate"><span class="pre">alias()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.any_match"><code class="docutils literal notranslate"><span class="pre">any_match()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.any_value"><code class="docutils literal notranslate"><span class="pre">any_value()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.approx_distinct"><code class="docutils literal notranslate"><span class="pre">approx_distinct()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.approx_median"><code class="docutils literal notranslate"><span class="pre">approx_median()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.approx_percentile_cont"><code class="docutils literal notranslate"><span class="pre">approx_percentile_cont()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.approx_percentile_cont_with_weight"><code class="docutils literal notranslate"><span class="pre">approx_percentile_cont_with_weight()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array"><code class="docutils literal notranslate"><span class="pre">array()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_add"><code class="docutils literal notranslate"><span class="pre">array_add()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_agg"><code class="docutils literal notranslate"><span class="pre">array_agg()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_any_match"><code class="docutils literal notranslate"><span class="pre">array_any_match()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_any_value"><code class="docutils literal notranslate"><span class="pre">array_any_value()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_append"><code class="docutils literal notranslate"><span class="pre">array_append()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_avg"><code class="docutils literal notranslate"><span class="pre">array_avg()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_cat"><code class="docutils literal notranslate"><span class="pre">array_cat()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_compact"><code class="docutils literal notranslate"><span class="pre">array_compact()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_concat"><code class="docutils literal notranslate"><span class="pre">array_concat()</span></code></a></li>
@@ -8142,6 +8659,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_except"><code class="docutils literal notranslate"><span class="pre">array_except()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_extract"><code class="docutils literal notranslate"><span class="pre">array_extract()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_filter"><code class="docutils literal notranslate"><span class="pre">array_filter()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_first"><code class="docutils literal notranslate"><span class="pre">array_first()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_has"><code class="docutils literal notranslate"><span class="pre">array_has()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_has_all"><code class="docutils literal notranslate"><span class="pre">array_has_all()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_has_any"><code class="docutils literal notranslate"><span class="pre">array_has_any()</span></code></a></li>
@@ -8158,6 +8676,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_position"><code class="docutils literal notranslate"><span class="pre">array_position()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_positions"><code class="docutils literal notranslate"><span class="pre">array_positions()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_prepend"><code class="docutils literal notranslate"><span class="pre">array_prepend()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_product"><code class="docutils literal notranslate"><span class="pre">array_product()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_push_back"><code class="docutils literal notranslate"><span class="pre">array_push_back()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_push_front"><code class="docutils literal notranslate"><span class="pre">array_push_front()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_remove"><code class="docutils literal notranslate"><span class="pre">array_remove()</span></code></a></li>
@@ -8169,8 +8688,11 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_replace_n"><code class="docutils literal notranslate"><span class="pre">array_replace_n()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_resize"><code class="docutils literal notranslate"><span class="pre">array_resize()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_reverse"><code class="docutils literal notranslate"><span class="pre">array_reverse()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_scale"><code class="docutils literal notranslate"><span class="pre">array_scale()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_slice"><code class="docutils literal notranslate"><span class="pre">array_slice()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_sort"><code class="docutils literal notranslate"><span class="pre">array_sort()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_subtract"><code class="docutils literal notranslate"><span class="pre">array_subtract()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_sum"><code class="docutils literal notranslate"><span class="pre">array_sum()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_to_string"><code class="docutils literal notranslate"><span class="pre">array_to_string()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_transform"><code class="docutils literal notranslate"><span class="pre">array_transform()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.array_union"><code class="docutils literal notranslate"><span class="pre">array_union()</span></code></a></li>
@@ -8240,6 +8762,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.exp"><code class="docutils literal notranslate"><span class="pre">exp()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.extract"><code class="docutils literal notranslate"><span class="pre">extract()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.factorial"><code class="docutils literal notranslate"><span class="pre">factorial()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.file_row_index"><code class="docutils literal notranslate"><span class="pre">file_row_index()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.find_in_set"><code class="docutils literal notranslate"><span class="pre">find_in_set()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.first_value"><code class="docutils literal notranslate"><span class="pre">first_value()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.flatten"><code class="docutils literal notranslate"><span class="pre">flatten()</span></code></a></li>
@@ -8255,6 +8778,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.in_list"><code class="docutils literal notranslate"><span class="pre">in_list()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.initcap"><code class="docutils literal notranslate"><span class="pre">initcap()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.inner_product"><code class="docutils literal notranslate"><span class="pre">inner_product()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.input_file_name"><code class="docutils literal notranslate"><span class="pre">input_file_name()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.instr"><code class="docutils literal notranslate"><span class="pre">instr()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.is_nan"><code class="docutils literal notranslate"><span class="pre">is_nan()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.isnan"><code class="docutils literal notranslate"><span class="pre">isnan()</span></code></a></li>
@@ -8269,9 +8793,11 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.left"><code class="docutils literal notranslate"><span class="pre">left()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.length"><code class="docutils literal notranslate"><span class="pre">length()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.levenshtein"><code class="docutils literal notranslate"><span class="pre">levenshtein()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_add"><code class="docutils literal notranslate"><span class="pre">list_add()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_any_match"><code class="docutils literal notranslate"><span class="pre">list_any_match()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_any_value"><code class="docutils literal notranslate"><span class="pre">list_any_value()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_append"><code class="docutils literal notranslate"><span class="pre">list_append()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_avg"><code class="docutils literal notranslate"><span class="pre">list_avg()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_cat"><code class="docutils literal notranslate"><span class="pre">list_cat()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_compact"><code class="docutils literal notranslate"><span class="pre">list_compact()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_concat"><code class="docutils literal notranslate"><span class="pre">list_concat()</span></code></a></li>
@@ -8284,6 +8810,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_except"><code class="docutils literal notranslate"><span class="pre">list_except()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_extract"><code class="docutils literal notranslate"><span class="pre">list_extract()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_filter"><code class="docutils literal notranslate"><span class="pre">list_filter()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_first"><code class="docutils literal notranslate"><span class="pre">list_first()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_has"><code class="docutils literal notranslate"><span class="pre">list_has()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_has_all"><code class="docutils literal notranslate"><span class="pre">list_has_all()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_has_any"><code class="docutils literal notranslate"><span class="pre">list_has_any()</span></code></a></li>
@@ -8301,6 +8828,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_position"><code class="docutils literal notranslate"><span class="pre">list_position()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_positions"><code class="docutils literal notranslate"><span class="pre">list_positions()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_prepend"><code class="docutils literal notranslate"><span class="pre">list_prepend()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_product"><code class="docutils literal notranslate"><span class="pre">list_product()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_push_back"><code class="docutils literal notranslate"><span class="pre">list_push_back()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_push_front"><code class="docutils literal notranslate"><span class="pre">list_push_front()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_remove"><code class="docutils literal notranslate"><span class="pre">list_remove()</span></code></a></li>
@@ -8312,8 +8840,11 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_replace_n"><code class="docutils literal notranslate"><span class="pre">list_replace_n()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_resize"><code class="docutils literal notranslate"><span class="pre">list_resize()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_reverse"><code class="docutils literal notranslate"><span class="pre">list_reverse()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_scale"><code class="docutils literal notranslate"><span class="pre">list_scale()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_slice"><code class="docutils literal notranslate"><span class="pre">list_slice()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_sort"><code class="docutils literal notranslate"><span class="pre">list_sort()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_subtract"><code class="docutils literal notranslate"><span class="pre">list_subtract()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_sum"><code class="docutils literal notranslate"><span class="pre">list_sum()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_to_string"><code class="docutils literal notranslate"><span class="pre">list_to_string()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_transform"><code class="docutils literal notranslate"><span class="pre">list_transform()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.list_union"><code class="docutils literal notranslate"><span class="pre">list_union()</span></code></a></li>
@@ -8358,6 +8889,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.power"><code class="docutils literal notranslate"><span class="pre">power()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.quantile_cont"><code class="docutils literal notranslate"><span class="pre">quantile_cont()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.radians"><code class="docutils literal notranslate"><span class="pre">radians()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.rand"><code class="docutils literal notranslate"><span class="pre">rand()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.random"><code class="docutils literal notranslate"><span class="pre">random()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.range"><code class="docutils literal notranslate"><span class="pre">range()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.rank"><code class="docutils literal notranslate"><span class="pre">rank()</span></code></a></li>
@@ -8405,6 +8937,7 @@ unchanged. Empty keys raise <code class="xref py py-class docutils literal notra
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.substr"><code class="docutils literal notranslate"><span class="pre">substr()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.substr_index"><code class="docutils literal notranslate"><span class="pre">substr_index()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.substring"><code class="docutils literal notranslate"><span class="pre">substring()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.substring_index"><code class="docutils literal notranslate"><span class="pre">substring_index()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.sum"><code class="docutils literal notranslate"><span class="pre">sum()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.tan"><code class="docutils literal notranslate"><span class="pre">tan()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.tanh"><code class="docutils literal notranslate"><span class="pre">tanh()</span></code></a></li>
diff --git a/autoapi/datafusion/functions/spark/index.html b/autoapi/datafusion/functions/spark/index.html
index a7a7250c..5294d53b 100644
--- a/autoapi/datafusion/functions/spark/index.html
+++ b/autoapi/datafusion/functions/spark/index.html
@@ -563,39 +563,51 @@ names) before issuing SQL queries.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.ascii" title="datafusion.functions.spark.ascii"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ascii</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">ascii</span></code>: code point of the first character.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.avg" title="datafusion.functions.spark.avg"><code class="xref py py-obj docutils literal notranslate"><span class="pre">avg</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.atan2" title="datafusion.functions.spark.atan2"><code class="xref py py-obj docutils literal notranslate"><span class="pre">atan2</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">atan2</span></code>: angle in radians of the point <code class="docutils literal notranslate"><span class="pre">(col2,</span> <span class="pre">col1)</span></code>.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.avg" title="datafusion.functions.spark.avg"><code class="xref py py-obj docutils literal notranslate"><span class="pre">avg</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">avg</span></code>: returns the mean of a numeric column.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.base64" title="datafusion.functions.spark.base64"><code class="xref py py-obj docutils literal notranslate"><span class="pre">base64</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.base64" title="datafusion.functions.spark.base64"><code class="xref py py-obj docutils literal notranslate"><span class="pre">base64</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">base64</span></code>: encode binary as a base64 string.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bin" title="datafusion.functions.spark.bin"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bin</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bin" title="datafusion.functions.spark.bin"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bin</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">bin</span></code>: binary string representation of a long.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bit_count" title="datafusion.functions.spark.bit_count"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bit_count</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bit_count" title="datafusion.functions.spark.bit_count"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bit_count</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">bit_count</span></code>: number of bits set in the integer's binary form.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bit_get" title="datafusion.functions.spark.bit_get"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bit_get</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bit_get" title="datafusion.functions.spark.bit_get"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bit_get</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">bit_get</span></code>: returns the bit (0 or 1) at <code class="docutils literal notranslate"><span class="pre">pos</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitmap_bit_position" title="datafusion.functions.spark.bitmap_bit_position"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitmap_bit_position</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitmap_bit_position" title="datafusion.functions.spark.bitmap_bit_position"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitmap_bit_position</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">bitmap_bit_position</span></code>: bit position for a child expression.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitmap_bucket_number" title="datafusion.functions.spark.bitmap_bucket_number"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitmap_bucket_number</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitmap_bucket_number" title="datafusion.functions.spark.bitmap_bucket_number"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitmap_bucket_number</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">bitmap_bucket_number</span></code>: bucket number for a child expression.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitmap_count" title="datafusion.functions.spark.bitmap_count"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitmap_count</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitmap_count" title="datafusion.functions.spark.bitmap_count"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitmap_count</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">bitmap_count</span></code>: number of set bits in a bitmap.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitwise_not" title="datafusion.functions.spark.bitwise_not"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitwise_not</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.bitwise_not" title="datafusion.functions.spark.bitwise_not"><code class="xref py py-obj docutils literal notranslate"><span class="pre">bitwise_not</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">~</span></code>: bitwise NOT.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.ceil" title="datafusion.functions.spark.ceil"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ceil</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.ceil" title="datafusion.functions.spark.ceil"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ceil</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">ceil</span></code>: smallest integer ≥ arg.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.ceiling" title="datafusion.functions.spark.ceiling"><code class="xref py py-obj docutils literal notranslate"><span class="pre">ceiling</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">ceiling</span></code>: smallest integer ≥ arg.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.char" title="datafusion.functions.spark.char"><code class="xref py py-obj docutils literal notranslate"><span class="pre">char</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">char</span></code>: ASCII character for a code point (mod 256).</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.char_length" title="datafusion.functions.spark.char_length"><code class="xref py py-obj docutils literal notranslate"><span class="pre">char_length</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">char_length</span></code>: character length of a string, or bytes of binary.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.character_length" title="datafusion.functions.spark.character_length"><code class="xref py py-obj docutils literal notranslate"><span class="pre">character_length</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">character_length</span></code>: character length of a string, or bytes of binary.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.collect_list" title="datafusion.functions.spark.collect_list"><code class="xref py py-obj docutils literal notranslate"><span class="pre">collect_list</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">collect_list</span></code>: collect values into an array (preserves dups).</p></td>
</tr>
@@ -605,27 +617,39 @@ names) before issuing SQL queries.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.concat" title="datafusion.functions.spark.concat"><code class="xref py py-obj docutils literal notranslate"><span class="pre">concat</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">concat</span></code>: concatenates strings; NULL if any input is NULL.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.crc32" title="datafusion.functions.spark.crc32"><code class="xref py py-obj docutils literal notranslate"><span class="pre">crc32</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.concat_ws" title="datafusion.functions.spark.concat_ws"><code class="xref py py-obj docutils literal notranslate"><span class="pre">concat_ws</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">concat_ws</span></code>: joins strings and arrays of strings with <code class="docutils literal notranslate"><span class="pre">sep</span></code>.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.crc32" title="datafusion.functions.spark.crc32"><code class="xref py py-obj docutils literal notranslate"><span class="pre">crc32</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">crc32</span></code>: cyclic redundancy check value as a bigint.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.csc" title="datafusion.functions.spark.csc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">csc</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.csc" title="datafusion.functions.spark.csc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">csc</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">csc</span></code>: cosecant.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_add" title="datafusion.functions.spark.date_add"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_add</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_add" title="datafusion.functions.spark.date_add"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_add</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">date_add</span></code>: date + N days.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_diff" title="datafusion.functions.spark.date_diff"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_diff</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_diff" title="datafusion.functions.spark.date_diff"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_diff</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">date_diff</span></code>: number of days from <code class="docutils literal notranslate"><span class="pre">start_date</span></code> to <code class="docutils literal notranslate"><span class="pre">end_date</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_part" title="datafusion.functions.spark.date_part"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_part</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_part" title="datafusion.functions.spark.date_part"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_part</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">date_part</span></code>: extract <code class="docutils literal notranslate"><span class="pre">field</span></code> from a date/time/timestamp.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_sub" title="datafusion.functions.spark.date_sub"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_sub</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_sub" title="datafusion.functions.spark.date_sub"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_sub</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">date_sub</span></code>: date - N days.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_trunc" title="datafusion.functions.spark.date_trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.date_trunc" title="datafusion.functions.spark.date_trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">date_trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">date_trunc</span></code>: truncate timestamp to unit <code class="docutils literal notranslate"><span class="pre">fmt</span></code>.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.dateadd" title="datafusion.functions.spark.dateadd"><code class="xref py py-obj docutils literal notranslate"><span class="pre">dateadd</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">dateadd</span></code>: date + N days.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.datediff" title="datafusion.functions.spark.datediff"><code class="xref py py-obj docutils literal notranslate"><span class="pre">datediff</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">datediff</span></code>: number of days from <code class="docutils literal notranslate"><span class="pre">start</span></code> to <code class="docutils literal notranslate"><span class="pre">end</span></code>.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.datepart" title="datafusion.functions.spark.datepart"><code class="xref py py-obj docutils literal notranslate"><span class="pre">datepart</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">datepart</span></code>: extract <code class="docutils literal notranslate"><span class="pre">field</span></code> from a date/time/timestamp.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.elt" title="datafusion.functions.spark.elt"><code class="xref py py-obj docutils literal notranslate"><span class="pre">elt</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">elt</span></code>: returns the n-th input (1-indexed).</p></td>
</tr>
@@ -644,12 +668,18 @@ names) before issuing SQL queries.</p>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.from_utc_timestamp" title="datafusion.functions.spark.from_utc_timestamp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">from_utc_timestamp</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">from_utc_timestamp</span></code>: interpret <code class="docutils literal notranslate"><span class="pre">ts</span></code> as UTC, convert to <code class="docutils literal notranslate"><span class="pre">tz</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.hex" title="datafusion.functions.spark.hex"><code class="xref py py-obj docutils literal notranslate"><span class="pre">hex</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.getbit" title="datafusion.functions.spark.getbit"><code class="xref py py-obj docutils literal notranslate"><span class="pre">getbit</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">getbit</span></code>: returns the bit (0 or 1) at <code class="docutils literal notranslate"><span class="pre">pos</span></code>.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.hex" title="datafusion.functions.spark.hex"><code class="xref py py-obj docutils literal notranslate"><span class="pre">hex</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">hex</span></code>: hexadecimal representation.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.hour" title="datafusion.functions.spark.hour"><code class="xref py py-obj docutils literal notranslate"><span class="pre">hour</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.hour" title="datafusion.functions.spark.hour"><code class="xref py py-obj docutils literal notranslate"><span class="pre">hour</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">hour</span></code>: extract hour component of a timestamp.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.hypot" title="datafusion.functions.spark.hypot"><code class="xref py py-obj docutils literal notranslate"><span class="pre">hypot</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">hypot</span></code>: <code class="docutils literal notranslate"><span class="pre">sqrt(col1^2</span> <span class="pre">+</span> <span class="pre">col2^2)</span></code> without intermediate overflow.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.if_" title="datafusion.functions.spark.if_"><code class="xref py py-obj docutils literal notranslate"><span class="pre">if_</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">if</span></code>: returns <code class="docutils literal notranslate"><span class="pre">if_true</span></code> when <code class="docutils literal notranslate"><span class="pre">condition</span></code> is true, else <code class="docutils literal notranslate"><span class="pre">if_false</span></code>.</p></td>
</tr>
@@ -695,30 +725,48 @@ names) before issuing SQL queries.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.modulus" title="datafusion.functions.spark.modulus"><code class="xref py py-obj docutils literal notranslate"><span class="pre">modulus</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">mod</span></code>: remainder of <code class="docutils literal notranslate"><span class="pre">dividend</span> <span class="pre">/</span> <span class="pre">divisor</span></code> (sign follows dividend).</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.negative" title="datafusion.functions.spark.negative"><code class="xref py py-obj docutils literal notranslate"><span class="pre">negative</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.monthname" title="datafusion.functions.spark.monthname"><code class="xref py py-obj docutils literal notranslate"><span class="pre">monthname</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">monthname</span></code>: three-letter abbreviated month name.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.negative" title="datafusion.functions.spark.negative"><code class="xref py py-obj docutils literal notranslate"><span class="pre">negative</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">negative</span></code>: unary minus.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.next_day" title="datafusion.functions.spark.next_day"><code class="xref py py-obj docutils literal notranslate"><span class="pre">next_day</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.next_day" title="datafusion.functions.spark.next_day"><code class="xref py py-obj docutils literal notranslate"><span class="pre">next_day</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">next_day</span></code>: first date after <code class="docutils literal notranslate"><span class="pre">start_date</span></code> named <code class="docutils literal notranslate"><span class="pre">day_of_week</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.parse_url" title="datafusion.functions.spark.parse_url"><code class="xref py py-obj docutils literal notranslate"><span class="pre">parse_url</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.parse_url" title="datafusion.functions.spark.parse_url"><code class="xref py py-obj docutils literal notranslate"><span class="pre">parse_url</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">parse_url</span></code>: extract a part from a URL; errors on invalid URLs.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.pmod" title="datafusion.functions.spark.pmod"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pmod</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.pmod" title="datafusion.functions.spark.pmod"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pmod</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">pmod</span></code>: positive remainder of division.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.rint" title="datafusion.functions.spark.rint"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rint</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.pow" title="datafusion.functions.spark.pow"><code class="xref py py-obj docutils literal notranslate"><span class="pre">pow</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">pow</span></code>: <code class="docutils literal notranslate"><span class="pre">col1</span></code> raised to the power <code class="docutils literal notranslate"><span class="pre">col2</span></code>, as a double.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.power" title="datafusion.functions.spark.power"><code class="xref py py-obj docutils literal notranslate"><span class="pre">power</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">power</span></code>: <code class="docutils literal notranslate"><span class="pre">col1</span></code> raised to the power <code class="docutils literal notranslate"><span class="pre">col2</span></code>.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.printf" title="datafusion.functions.spark.printf"><code class="xref py py-obj docutils literal notranslate"><span class="pre">printf</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">printf</span></code>: printf-style format string.</p></td>
+</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.quote" title="datafusion.functions.spark.quote"><code class="xref py py-obj docutils literal notranslate"><span class="pre">quote</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">quote</span></code>: wraps a string in single quotes, escaping inner quotes.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.rint" title="datafusion.functions.spark.rint"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rint</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">rint</span></code>: round to nearest mathematical integer (as double).</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.round" title="datafusion.functions.spark.round"><code class="xref py py-obj docutils literal notranslate"><span class="pre">round</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.round" title="datafusion.functions.spark.round"><code class="xref py py-obj docutils literal notranslate"><span class="pre">round</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">round</span></code>: round to <code class="docutils literal notranslate"><span class="pre">scale</span></code> decimal places, HALF_UP rounding.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.sec" title="datafusion.functions.spark.sec"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sec</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.sec" title="datafusion.functions.spark.sec"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sec</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">sec</span></code>: secant.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.second" title="datafusion.functions.spark.second"><code class="xref py py-obj docutils literal notranslate"><span class="pre">second</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.second" title="datafusion.functions.spark.second"><code class="xref py py-obj docutils literal notranslate"><span class="pre">second</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">second</span></code>: extract second component of a timestamp.</p></td>
</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.sha" title="datafusion.functions.spark.sha"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sha</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">sha</span></code>: SHA-1 hash as a hex string.</p></td>
+</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.sha1" title="datafusion.functions.spark.sha1"><code class="xref py py-obj docutils literal notranslate"><span class="pre">sha1</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">sha1</span></code>: SHA-1 hash as a hex string.</p></td>
</tr>
@@ -755,51 +803,57 @@ names) before issuing SQL queries.</p>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.str_to_map" title="datafusion.functions.spark.str_to_map"><code class="xref py py-obj docutils literal notranslate"><span class="pre">str_to_map</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">str_to_map</span></code>: split text into key/value pairs using delimiters.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.substring" title="datafusion.functions.spark.substring"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substring</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.substr" title="datafusion.functions.spark.substr"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substr</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">substr</span></code>: 1-indexed substring, to the end when <code class="docutils literal notranslate"><span class="pre">len</span></code> is omitted.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.substring" title="datafusion.functions.spark.substring"><code class="xref py py-obj docutils literal notranslate"><span class="pre">substring</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">substring</span></code>: 1-indexed substring starting at <code class="docutils literal notranslate"><span class="pre">pos</span></code> of given <code class="docutils literal notranslate"><span class="pre">length</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.time_trunc" title="datafusion.functions.spark.time_trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">time_trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.time_trunc" title="datafusion.functions.spark.time_trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">time_trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">time_trunc</span></code>: truncate time value to unit <code class="docutils literal notranslate"><span class="pre">fmt</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.to_utc_timestamp" title="datafusion.functions.spark.to_utc_timestamp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_utc_timestamp</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.to_utc_timestamp" title="datafusion.functions.spark.to_utc_timestamp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">to_utc_timestamp</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">to_utc_timestamp</span></code>: interpret <code class="docutils literal notranslate"><span class="pre">ts</span></code> as <code class="docutils literal notranslate"><span class="pre">tz</span></code>, convert to UTC.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.trunc" title="datafusion.functions.spark.trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.trunc" title="datafusion.functions.spark.trunc"><code class="xref py py-obj docutils literal notranslate"><span class="pre">trunc</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">trunc</span></code>: truncate date to unit <code class="docutils literal notranslate"><span class="pre">fmt</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.try_parse_url" title="datafusion.functions.spark.try_parse_url"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_parse_url</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.try_parse_url" title="datafusion.functions.spark.try_parse_url"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_parse_url</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">try_parse_url</span></code>: like <code class="docutils literal notranslate"><span class="pre">parse_url</span></code> but returns NULL on invalid URLs.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.try_sum" title="datafusion.functions.spark.try_sum"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_sum</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.try_sum" title="datafusion.functions.spark.try_sum"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_sum</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">try_sum</span></code>: sum that returns NULL on overflow.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.try_url_decode" title="datafusion.functions.spark.try_url_decode"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_url_decode</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.try_url_decode" title="datafusion.functions.spark.try_url_decode"><code class="xref py py-obj docutils literal notranslate"><span class="pre">try_url_decode</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">try_url_decode</span></code>: like <code class="docutils literal notranslate"><span class="pre">url_decode</span></code>; returns NULL on invalid input.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.unbase64" title="datafusion.functions.spark.unbase64"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unbase64</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.unbase64" title="datafusion.functions.spark.unbase64"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unbase64</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">unbase64</span></code>: decode a base64 string to binary.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.unhex" title="datafusion.functions.spark.unhex"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unhex</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.unhex" title="datafusion.functions.spark.unhex"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unhex</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">unhex</span></code>: convert hexadecimal string to binary.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_date" title="datafusion.functions.spark.unix_date"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_date</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_date" title="datafusion.functions.spark.unix_date"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_date</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">unix_date</span></code>: days since 1970-01-01 for <code class="docutils literal notranslate"><span class="pre">dt</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_micros" title="datafusion.functions.spark.unix_micros"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_micros</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_micros" title="datafusion.functions.spark.unix_micros"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_micros</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">unix_micros</span></code>: microseconds since epoch for <code class="docutils literal notranslate"><span class="pre">ts</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_millis" title="datafusion.functions.spark.unix_millis"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_millis</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_millis" title="datafusion.functions.spark.unix_millis"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_millis</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">unix_millis</span></code>: milliseconds since epoch for <code class="docutils literal notranslate"><span class="pre">ts</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_seconds" title="datafusion.functions.spark.unix_seconds"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_seconds</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.unix_seconds" title="datafusion.functions.spark.unix_seconds"><code class="xref py py-obj docutils literal notranslate"><span class="pre">unix_seconds</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">unix_seconds</span></code>: seconds since epoch for <code class="docutils literal notranslate"><span class="pre">ts</span></code>.</p></td>
</tr>
-<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.url_decode" title="datafusion.functions.spark.url_decode"><code class="xref py py-obj docutils literal notranslate"><span class="pre">url_decode</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.url_decode" title="datafusion.functions.spark.url_decode"><code class="xref py py-obj docutils literal notranslate"><span class="pre">url_decode</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">url_decode</span></code>: decode an application/x-www-form-urlencoded string.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.url_encode" title="datafusion.functions.spark.url_encode"><code class="xref py py-obj docutils literal notranslate"><span class="pre">url_encode</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.url_encode" title="datafusion.functions.spark.url_encode"><code class="xref py py-obj docutils literal notranslate"><span class="pre">url_encode</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">url_encode</span></code>: encode a string in application/x-www-form-urlencoded.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.functions.spark.weekday" title="datafusion.functions.spark.weekday"><code class="xref py py-obj docutils literal notranslate"><span class="pre">weekday</span></code></a>(→ datafusion.expr.Expr)</p></td>
+<td><p>Spark <code class="docutils literal notranslate"><span class="pre">weekday</span></code>: day of the week, Monday = 0 through Sunday = 6.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.functions.spark.width_bucket" title="datafusion.functions.spark.width_bucket"><code class="xref py py-obj docutils literal notranslate"><span class="pre">width_bucket</span></code></a>(→ datafusion.expr.Expr)</p></td>
<td><p>Spark <code class="docutils literal notranslate"><span class="pre">width_bucket</span></code>: bucket number for <code class="docutils literal notranslate"><span class="pre">value</span></code> in equi-width histogram.</p></td>
</tr>
@@ -911,6 +965,22 @@ names) before issuing SQL queries.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.atan2">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">atan2</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">atan2</span></code>: angle in radians of the point <code class="docutils literal notranslate"><span class="pre">(col2,</span> <span class="pre">col1)</span></code>.</p>
+<p><code class="docutils literal notranslate"><span class="pre">col1</span></code> is the y coordinate and <code class="docutils literal notranslate"><span class="pre">col2</span></code> the x coordinate. Both accept
+native numbers or <code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code>.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">atan2</span><span class="p">(</span><span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">1.5707963267948966</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.avg">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">avg</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
@@ -1058,6 +1128,16 @@ not a literal; pass <a class="reference internal" href="../../index.html#datafus
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.ceiling">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">ceiling</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">ceiling</span></code>: smallest integer ≥ arg.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.ceil" title="datafusion.functions.spark.ceil"><code class="xref py py-func docutils literal notranslate"><span class="pre">ceil()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.char">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">char</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
@@ -1072,6 +1152,26 @@ not a literal; pass <a class="reference internal" href="../../index.html#datafus
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.char_length">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">char_length</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pr [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">char_length</span></code>: character length of a string, or bytes of binary.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.length" title="datafusion.functions.spark.length"><code class="xref py py-func docutils literal notranslate"><span class="pre">length()</span></code></a>.</p>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.character_length">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">character_length</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span clas [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">character_length</span></code>: character length of a string, or bytes of binary.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.length" title="datafusion.functions.spark.length"><code class="xref py py-func docutils literal notranslate"><span class="pre">length()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.collect_list">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">collect_list</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="p [...]
@@ -1117,6 +1217,25 @@ not a literal; pass <a class="reference internal" href="../../index.html#datafus
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.concat_ws">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">concat_ws</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">sep</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">concat_ws</span></code>: joins strings and arrays of strings with <code class="docutils literal notranslate"><span class="pre">sep</span></code>.</p>
+<p>NULL inputs are skipped rather than propagated.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"x"</span><span class="p">],</span> <span class="s2">"b"</span><span class="p">:</span> <span class="p">[</span><span class="kc">None</span><span class="p">],</span> <span [...]
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">concat_ws</span><span class="p">(</span>
+<span class="gp">... </span> <span class="s2">"-"</span><span class="p">,</span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span> <span class="n">dfn</span><span class="o">.</span><span class="n">co [...]
+<span class="gp">... </span> <span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)</span>
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'x-z'</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.crc32">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">crc32</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
@@ -1239,6 +1358,36 @@ not a literal; pass <a class="reference internal" href="../../index.html#datafus
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.dateadd">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">dateadd</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">start</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">dateadd</span></code>: date + N days.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.date_add" title="datafusion.functions.spark.date_add"><code class="xref py py-func docutils literal notranslate"><span class="pre">date_add()</span></code></a>.</p>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.datediff">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">datediff</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">end</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">datediff</span></code>: number of days from <code class="docutils literal notranslate"><span class="pre">start</span></code> to <code class="docutils literal notranslate"><span class="pre">end</span></code>.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.date_diff" title="datafusion.functions.spark.date_diff"><code class="xref py py-func docutils literal notranslate"><span class="pre">date_diff()</span></code></a>.</p>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.datepart">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">datepart</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">field</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">datepart</span></code>: extract <code class="docutils literal notranslate"><span class="pre">field</span></code> from a date/time/timestamp.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.date_part" title="datafusion.functions.spark.date_part"><code class="xref py py-func docutils literal notranslate"><span class="pre">date_part()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.elt">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">elt</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">inputs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" [...]
@@ -1345,6 +1494,16 @@ to a literal expression); remaining args are values to substitute.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.getbit">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">getbit</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">getbit</span></code>: returns the bit (0 or 1) at <code class="docutils literal notranslate"><span class="pre">pos</span></code>.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.bit_get" title="datafusion.functions.spark.bit_get"><code class="xref py py-func docutils literal notranslate"><span class="pre">bit_get()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.hex">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">hex</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
@@ -1378,6 +1537,21 @@ to a literal expression); remaining args are values to substitute.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.hypot">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">hypot</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">hypot</span></code>: <code class="docutils literal notranslate"><span class="pre">sqrt(col1^2</span> <span class="pre">+</span> <span class="pre">col2^2)</span></code> without intermediate overflow.</p>
+<p>Both arguments accept native numbers or <code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code>.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">hypot</span><span class="p">(</span><span class="mf">3.0</span><span class="p">,</span> <span class="mf">4.0</span><span class="p [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">5.0</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.if_">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">if_</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">condition</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
@@ -1453,7 +1627,7 @@ the Rust binding; passing a non-<code class="docutils literal notranslate"><span
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.last_day">
-<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">last_day</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">last_day</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">date</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">last_day</span></code>: last day of the month containing the date.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span>
@@ -1657,6 +1831,23 @@ field becomes the map key and the second the value.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.monthname">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">monthname</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">monthname</span></code>: three-letter abbreviated month name.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datetime</span><span class="w"> </span><span class="kn">import</span> <span class="n">date</span>
+<span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">d</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="n">pa</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="n">date</span><span class="p">(</span><span class="mi">2024</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">15</span><span class="p">)))</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">monthname</span><span class="p">(</span><span class="n">d</span><span class="p">)</span><span class="o">.</span><span class="n">a [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'Mar'</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.negative">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">negative</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre"> [...]
@@ -1739,6 +1930,68 @@ Pass <code class="docutils literal notranslate"><span class="pre">key</span></co
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.pow">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">pow</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">pow</span></code>: <code class="docutils literal notranslate"><span class="pre">col1</span></code> raised to the power <code class="docutils literal notranslate"><span class="pre">col2</span></code>, as a double.</p>
+<p>Both arguments accept native numbers or <code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code>.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">10</span><span class="p">)</ [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">1024.0</span>
+</pre></div>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.power">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">power</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col1</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">power</span></code>: <code class="docutils literal notranslate"><span class="pre">col1</span></code> raised to the power <code class="docutils literal notranslate"><span class="pre">col2</span></code>.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.pow" title="datafusion.functions.spark.pow"><code class="xref py py-func docutils literal notranslate"><span class="pre">pow()</span></code></a>.</p>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.printf">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">printf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">format</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">printf</span></code>: printf-style format string.</p>
+<p>Unlike <a class="reference internal" href="#datafusion.functions.spark.format_string" title="datafusion.functions.spark.format_string"><code class="xref py py-func docutils literal notranslate"><span class="pre">format_string()</span></code></a>, a bare <code class="docutils literal notranslate"><span class="pre">str</span></code> <code class="docutils literal notranslate"><span class="pre">format</span></code> or <code class="docutils literal notranslate"><span class="pre">cols</span [...]
+is treated as a column name (matching pyspark), not a literal; pass
+<a class="reference internal" href="../../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">lit()</span></code></a> for a literal format.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"aa</span><span class="si">%d%s</span><span class="s2">"</span><span class="p">],</span> <span class="s2">"b"</span><span class="p">:</span> <span class="p">[</span><span c [...]
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">printf</span><span class="p">(</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b" [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'aa123cc'</span>
+</pre></div>
+</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">printf</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="s2">"</span><span class="si">%d</span><span class="s2">-</span><span class="si">%s</span><span class="s2">"</span><span class= [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'123-cc'</span>
+</pre></div>
+</div>
+</dd></dl>
+
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.quote">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">quote</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dat [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">quote</span></code>: wraps a string in single quotes, escaping inner quotes.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">( [...]
+<span class="gp">>>> </span><span class="nb">print</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">())</span>
+<span class="go">'it\'s'</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.rint">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">rint</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
@@ -1808,6 +2061,16 @@ native <code class="docutils literal notranslate"><span class="pre">int</span></
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.sha">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">sha</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">dataf [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">sha</span></code>: SHA-1 hash as a hex string.</p>
+<div class="admonition seealso">
+<p class="admonition-title">See also</p>
+<p>This is an alias for <a class="reference internal" href="#datafusion.functions.spark.sha1" title="datafusion.functions.spark.sha1"><code class="xref py py-func docutils literal notranslate"><span class="pre">sha1()</span></code></a>.</p>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.sha1">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">sha1</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">data [...]
@@ -2024,6 +2287,28 @@ name. Pass <a class="reference internal" href="../../index.html#datafusion.lit"
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.substr">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">substr</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">str</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">da [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">substr</span></code>: 1-indexed substring, to the end when <code class="docutils literal notranslate"><span class="pre">len</span></code> is omitted.</p>
+<p>Same as <a class="reference internal" href="#datafusion.functions.spark.substring" title="datafusion.functions.spark.substring"><code class="xref py py-func docutils literal notranslate"><span class="pre">substring()</span></code></a> except that <code class="docutils literal notranslate"><span class="pre">len</span></code> is optional. <code class="docutils literal notranslate"><span class="pre">pos</span></code> and
+<code class="docutils literal notranslate"><span class="pre">len</span></code> accept native <code class="docutils literal notranslate"><span class="pre">int</span></code> values or <code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code>.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">substr</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p"> [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'ello'</span>
+</pre></div>
+</div>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span>
+<span class="gp">... </span> <span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">substr</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="s2">"hello"</span><span class="p">),</span> <span class="mi">2</span><span class="p">,</span> <span class="nb">len</span><span [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">'ell'</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.substring">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">substring</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">str</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre" [...]
@@ -2283,6 +2568,23 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.functions.spark.weekday">
+<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">weekday</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">col</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">d [...]
+<dd><p>Spark <code class="docutils literal notranslate"><span class="pre">weekday</span></code>: day of the week, Monday = 0 through Sunday = 6.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datetime</span><span class="w"> </span><span class="kn">import</span> <span class="n">date</span>
+<span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"x"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">]})</span>
+<span class="gp">>>> </span><span class="n">d</span> <span class="o">=</span> <span class="n">dfn</span><span class="o">.</span><span class="n">lit</span><span class="p">(</span><span class="n">pa</span><span class="o">.</span><span class="n">scalar</span><span class="p">(</span><span class="n">date</span><span class="p">(</span><span class="mi">2024</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">15</span><span class="p">)))</span>
+<span class="gp">>>> </span><span class="n">r</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">dfn</span><span class="o">.</span><span class="n">functions</span><span class="o">.</span><span class="n">spark</span><span class="o">.</span><span class="n">weekday</span><span class="p">(</span><span class="n">d</span><span class="p">)</span><span class="o">.</span><span class="n">ali [...]
+<span class="gp">>>> </span><span class="n">r</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">as_py</span><span class="p">()</span>
+<span class="go">4</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.functions.spark.width_bucket">
<span class="sig-prename descclassname"><span class="pre">datafusion.functions.spark.</span></span><span class="sig-name descname"><span class="pre">width_bucket</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">v</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre [...]
@@ -2375,6 +2677,7 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.array_contains"><code class="docutils literal notranslate"><span class="pre">array_contains()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.array_repeat"><code class="docutils literal notranslate"><span class="pre">array_repeat()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.ascii"><code class="docutils literal notranslate"><span class="pre">ascii()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.atan2"><code class="docutils literal notranslate"><span class="pre">atan2()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.avg"><code class="docutils literal notranslate"><span class="pre">avg()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.base64"><code class="docutils literal notranslate"><span class="pre">base64()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.bin"><code class="docutils literal notranslate"><span class="pre">bin()</span></code></a></li>
@@ -2385,10 +2688,14 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.bitmap_count"><code class="docutils literal notranslate"><span class="pre">bitmap_count()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.bitwise_not"><code class="docutils literal notranslate"><span class="pre">bitwise_not()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.ceil"><code class="docutils literal notranslate"><span class="pre">ceil()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.ceiling"><code class="docutils literal notranslate"><span class="pre">ceiling()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.char"><code class="docutils literal notranslate"><span class="pre">char()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.char_length"><code class="docutils literal notranslate"><span class="pre">char_length()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.character_length"><code class="docutils literal notranslate"><span class="pre">character_length()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.collect_list"><code class="docutils literal notranslate"><span class="pre">collect_list()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.collect_set"><code class="docutils literal notranslate"><span class="pre">collect_set()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.concat"><code class="docutils literal notranslate"><span class="pre">concat()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.concat_ws"><code class="docutils literal notranslate"><span class="pre">concat_ws()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.crc32"><code class="docutils literal notranslate"><span class="pre">crc32()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.csc"><code class="docutils literal notranslate"><span class="pre">csc()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.date_add"><code class="docutils literal notranslate"><span class="pre">date_add()</span></code></a></li>
@@ -2396,14 +2703,19 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.date_part"><code class="docutils literal notranslate"><span class="pre">date_part()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.date_sub"><code class="docutils literal notranslate"><span class="pre">date_sub()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.date_trunc"><code class="docutils literal notranslate"><span class="pre">date_trunc()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.dateadd"><code class="docutils literal notranslate"><span class="pre">dateadd()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.datediff"><code class="docutils literal notranslate"><span class="pre">datediff()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.datepart"><code class="docutils literal notranslate"><span class="pre">datepart()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.elt"><code class="docutils literal notranslate"><span class="pre">elt()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.expm1"><code class="docutils literal notranslate"><span class="pre">expm1()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.factorial"><code class="docutils literal notranslate"><span class="pre">factorial()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.floor"><code class="docutils literal notranslate"><span class="pre">floor()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.format_string"><code class="docutils literal notranslate"><span class="pre">format_string()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.from_utc_timestamp"><code class="docutils literal notranslate"><span class="pre">from_utc_timestamp()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.getbit"><code class="docutils literal notranslate"><span class="pre">getbit()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.hex"><code class="docutils literal notranslate"><span class="pre">hex()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.hour"><code class="docutils literal notranslate"><span class="pre">hour()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.hypot"><code class="docutils literal notranslate"><span class="pre">hypot()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.if_"><code class="docutils literal notranslate"><span class="pre">if_()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.ilike"><code class="docutils literal notranslate"><span class="pre">ilike()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.is_valid_utf8"><code class="docutils literal notranslate"><span class="pre">is_valid_utf8()</span></code></a></li>
@@ -2419,14 +2731,20 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.map_from_entries"><code class="docutils literal notranslate"><span class="pre">map_from_entries()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.minute"><code class="docutils literal notranslate"><span class="pre">minute()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.modulus"><code class="docutils literal notranslate"><span class="pre">modulus()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.monthname"><code class="docutils literal notranslate"><span class="pre">monthname()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.negative"><code class="docutils literal notranslate"><span class="pre">negative()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.next_day"><code class="docutils literal notranslate"><span class="pre">next_day()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.parse_url"><code class="docutils literal notranslate"><span class="pre">parse_url()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.pmod"><code class="docutils literal notranslate"><span class="pre">pmod()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.pow"><code class="docutils literal notranslate"><span class="pre">pow()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.power"><code class="docutils literal notranslate"><span class="pre">power()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.printf"><code class="docutils literal notranslate"><span class="pre">printf()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.quote"><code class="docutils literal notranslate"><span class="pre">quote()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.rint"><code class="docutils literal notranslate"><span class="pre">rint()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.round"><code class="docutils literal notranslate"><span class="pre">round()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.sec"><code class="docutils literal notranslate"><span class="pre">sec()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.second"><code class="docutils literal notranslate"><span class="pre">second()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.sha"><code class="docutils literal notranslate"><span class="pre">sha()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.sha1"><code class="docutils literal notranslate"><span class="pre">sha1()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.sha2"><code class="docutils literal notranslate"><span class="pre">sha2()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.shiftleft"><code class="docutils literal notranslate"><span class="pre">shiftleft()</span></code></a></li>
@@ -2439,6 +2757,7 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.space"><code class="docutils literal notranslate"><span class="pre">space()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.spark_cast"><code class="docutils literal notranslate"><span class="pre">spark_cast()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.str_to_map"><code class="docutils literal notranslate"><span class="pre">str_to_map()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.substr"><code class="docutils literal notranslate"><span class="pre">substr()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.substring"><code class="docutils literal notranslate"><span class="pre">substring()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.time_trunc"><code class="docutils literal notranslate"><span class="pre">time_trunc()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.to_utc_timestamp"><code class="docutils literal notranslate"><span class="pre">to_utc_timestamp()</span></code></a></li>
@@ -2454,6 +2773,7 @@ names (matching pyspark); pass <a class="reference internal" href="../../index.h
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.unix_seconds"><code class="docutils literal notranslate"><span class="pre">unix_seconds()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.url_decode"><code class="docutils literal notranslate"><span class="pre">url_decode()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.url_encode"><code class="docutils literal notranslate"><span class="pre">url_encode()</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.weekday"><code class="docutils literal notranslate"><span class="pre">weekday()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.width_bucket"><code class="docutils literal notranslate"><span class="pre">width_bucket()</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#datafusion.functions.spark.xxhash64"><code class="docutils literal notranslate"><span class="pre">xxhash64()</span></code></a></li>
</ul>
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index 6668fe7a..c3398277 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -1633,8 +1633,9 @@ through <code class="docutils literal notranslate"><span class="pre">pickle.dump
<span class="sig-name descname"><span class="pre">distinct</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">ExprFuncBuilder</span></span></span><a class="headerlink" href="#datafusion.Expr.distinct" title="Link to this definition">#</a></dt>
<dd><p>Only evaluate distinct values for an aggregate function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on a window
+function unless that is an aggregate used as one. If used on any other type of
+expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py method">
@@ -1672,8 +1673,9 @@ type of expression, an error will be generated when <code class="docutils litera
<span class="sig-name descname"><span class="pre">filter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">filter</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.Expr" title="datafusion.Expr"><span class="pre">Expr</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span [...]
<dd><p>Filter an aggregate function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on a window
+function unless that is an aggregate used as one. If used on any other type of
+expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py method">
@@ -1898,10 +1900,15 @@ type of expression, an error will be generated when <code class="docutils litera
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.Expr.over">
<span class="sig-name descname"><span class="pre">over</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">window</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="expr/index.html#datafusion.expr.Window" title="datafusion.expr.Window"><span class="pre">Window</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="si [...]
-<dd><p>Turn an aggregate function into a window function.</p>
-<p>This function turns any aggregate function into a window function. With the
+<dd><p>Evaluate an aggregate or window function over a window.</p>
+<p>On an aggregate function this turns it into a window function. With the
exception of <code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, how each of the parameters is used is determined
by the underlying aggregate function.</p>
+<p>On an aggregate, the <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, <code class="docutils literal notranslate"><span class="pre">filter</span></code>, and <code class="docutils literal notranslate"><span class="pre">distinct</span></code> options
+it was built with are kept, and an <code class="docutils literal notranslate"><span class="pre">order_by</span></code> raises; see
+<a class="reference internal" href="../../user-guide/common-operations/windows.html#aggregate-over-options"><span class="std std-ref">Options set on the aggregate</span></a>.</p>
+<p>On a window function that already has a <code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, or
+<code class="docutils literal notranslate"><span class="pre">window_frame</span></code>, this raises; see <a class="reference internal" href="../../user-guide/common-operations/windows.html#window-function-chaining"><span class="std std-ref">Chaining onto a window function</span></a>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>window</strong> – Window definition</p>
@@ -1914,8 +1921,10 @@ by the underlying aggregate function.</p>
<span class="sig-name descname"><span class="pre">partition_by</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">partition_by</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.Expr" title="datafusion.Expr"><span class="pre">Expr</span></a></span></em><span class="sig-paren">)</span> <span cl [...]
<dd><p>Set the partitioning for a window function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on an
+aggregate function that has not been turned into a window function with
+<a class="reference internal" href="#datafusion.Expr.over" title="datafusion.Expr.over"><code class="xref py py-meth docutils literal notranslate"><span class="pre">over()</span></code></a>. If used on any other type of expression, an error will be
+generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py method">
@@ -2206,10 +2215,12 @@ SqlType Enum which this expression represents.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.Expr.window_frame">
<span class="sig-name descname"><span class="pre">window_frame</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">window_frame</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.WindowFrame" title="datafusion.WindowFrame"><span class="pre">WindowFrame</span></a></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class [...]
-<dd><p>Set the frame fora window function.</p>
+<dd><p>Set the frame for a window function.</p>
<p>This function will create an <code class="xref py py-class docutils literal notranslate"><span class="pre">ExprFuncBuilder</span></code> that can be used to
-set parameters for either window or aggregate functions. If used on any other
-type of expression, an error will be generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
+set parameters for either window or aggregate functions. It raises on an
+aggregate function that has not been turned into a window function with
+<a class="reference internal" href="#datafusion.Expr.over" title="datafusion.Expr.over"><code class="xref py py-meth docutils literal notranslate"><span class="pre">over()</span></code></a>. If used on any other type of expression, an error will be
+generated when <code class="docutils literal notranslate"><span class="pre">build()</span></code> is called.</p>
</dd></dl>
<dl class="py attribute">
diff --git a/autoapi/datafusion/user_defined/index.html b/autoapi/datafusion/user_defined/index.html
index e7b9e563..03ebbc35 100644
--- a/autoapi/datafusion/user_defined/index.html
+++ b/autoapi/datafusion/user_defined/index.html
@@ -601,7 +601,7 @@
<h2>Functions<a class="headerlink" href="#functions" title="Link to this heading">#</a></h2>
<div class="pst-scrollable-table-container"><table class="autosummary longtable table autosummary">
<tbody>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.user_defined._is_pycapsule" title="datafusion.user_defined._is_pycapsule"><code class="xref py py-obj docutils literal notranslate"><span class="pre">_is_pycapsule</span></code></a>(→ TypeGuard[_typeshed.CapsuleType])</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.user_defined._is_pycapsule" title="datafusion.user_defined._is_pycapsule"><code class="xref py py-obj docutils literal notranslate"><span class="pre">_is_pycapsule</span></code></a>(→ TypeGuard[types.CapsuleType])</p></td>
<td><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> when <code class="docutils literal notranslate"><span class="pre">value</span></code> is a CPython <code class="docutils literal notranslate"><span class="pre">PyCapsule</span></code>.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.user_defined._wrap_session_kwarg_for_udtf" title="datafusion.user_defined._wrap_session_kwarg_for_udtf"><code class="xref py py-obj docutils literal notranslate"><span class="pre">_wrap_session_kwarg_for_udtf</span></code></a>(...)</p></td>
@@ -692,14 +692,14 @@ occur during the evaluation of the dataframe.</p>
<dt class="sig sig-object py" id="datafusion.user_defined.AggregateUDF._from_internal">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_from_internal</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">internal</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.AggregateUDF</span></span></em><span class="sig-paren">)</span> <span class="sig-return">< [...]
<dd><p>Wrap an already-constructed internal <code class="docutils literal notranslate"><span class="pre">AggregateUDF</span></code> handle.</p>
-<p>Used by <code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.udaf()</span></code> to surface a function looked
-up from the session’s function registry without re-running
-<code class="xref py py-meth docutils literal notranslate"><span class="pre">__init__()</span></code>.</p>
+<p>Used by <code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.udaf()</span></code> and <a class="reference internal" href="#datafusion.user_defined.AggregateUDF.from_pycapsule" title="datafusion.user_defined.AggregateUDF.from_pycapsule"><code class="xref py py-meth docutils literal notranslate"><span class="pre">from_pycapsule()</span></code></a>
+to wrap a handle from the session’s function registry or an FFI
+capsule without re-running <code class="xref py py-meth docutils literal notranslate"><span class="pre">__init__()</span></code>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.AggregateUDF.from_pycapsule">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.AggregateUDFExportable [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.AggregateUDFExportable [...]
<dd><p>Create an Aggregate UDF from AggregateUDF PyCapsule object.</p>
<p>This function will instantiate a Aggregate UDF that uses a DataFusion
AggregateUDF that is exported via the FFI bindings.</p>
@@ -713,7 +713,7 @@ AggregateUDF that is exported via the FFI bindings.</p>
<dt class="sig sig-object py">
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.AggregateUDFExportable"><span c [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">_typeshed.CapsuleType</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">& [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">types.CapsuleType</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">! [...]
<dd><p>Create a new User-Defined Aggregate Function (UDAF).</p>
<p>This class allows you to define an aggregate function that can be used in
data aggregation or window function calls.</p>
@@ -943,14 +943,14 @@ occur during the evaluation of the dataframe.</p>
<dt class="sig sig-object py" id="datafusion.user_defined.ScalarUDF._from_internal">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_from_internal</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">internal</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.ScalarUDF</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><spa [...]
<dd><p>Wrap an already-constructed internal <code class="docutils literal notranslate"><span class="pre">ScalarUDF</span></code> handle.</p>
-<p>Used by <code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.udf()</span></code> to surface a function looked
-up from the session’s function registry without re-running
-<code class="xref py py-meth docutils literal notranslate"><span class="pre">__init__()</span></code>.</p>
+<p>Used by <code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.udf()</span></code> and <a class="reference internal" href="#datafusion.user_defined.ScalarUDF.from_pycapsule" title="datafusion.user_defined.ScalarUDF.from_pycapsule"><code class="xref py py-meth docutils literal notranslate"><span class="pre">from_pycapsule()</span></code></a>
+to wrap a handle from the session’s function registry or an FFI
+capsule without re-running <code class="xref py py-meth docutils literal notranslate"><span class="pre">__init__()</span></code>.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.ScalarUDF.from_pycapsule">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.ScalarUDFExportable" title="datafusion.user_defined.ScalarUDFExportable"><spa [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.ScalarUDFExportable" title="datafusion.user_defined.ScalarUDFExportable"><spa [...]
<dd><p>Create a Scalar UDF from ScalarUDF PyCapsule object.</p>
<p>This function will instantiate a Scalar UDF that uses a DataFusion
ScalarUDF that is exported via the FFI bindings.</p>
@@ -963,6 +963,8 @@ ScalarUDF that is exported via the FFI bindings.</p>
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><s [...]
<dt class="sig sig-object py">
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.ScalarUDFExportable" title="datafusion.user_defined.ScalarUDFExportable"><span class="pr [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">types.CapsuleType</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→ [...]
<dd><p>Create a new User-Defined Function (UDF).</p>
<p>This class can be used both as either a function or a decorator.</p>
<dl class="simple">
@@ -1400,9 +1402,9 @@ occur during the evaluation of the dataframe.</p>
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF._from_internal">
<em class="property"><span class="pre">classmethod</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_from_internal</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">internal</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.WindowUDF</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><spa [...]
<dd><p>Wrap an already-constructed internal <code class="docutils literal notranslate"><span class="pre">WindowUDF</span></code> handle.</p>
-<p>Used by <code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.udwf()</span></code> to surface a function looked
-up from the session’s function registry without re-running
-<code class="xref py py-meth docutils literal notranslate"><span class="pre">__init__()</span></code>.</p>
+<p>Used by <code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.udwf()</span></code> and <a class="reference internal" href="#datafusion.user_defined.WindowUDF.from_pycapsule" title="datafusion.user_defined.WindowUDF.from_pycapsule"><code class="xref py py-meth docutils literal notranslate"><span class="pre">from_pycapsule()</span></code></a>
+to wrap a handle from the session’s function registry or an FFI
+capsule without re-running <code class="xref py py-meth docutils literal notranslate"><span class="pre">__init__()</span></code>.</p>
</dd></dl>
<dl class="py method">
@@ -1419,7 +1421,7 @@ up from the session’s function registry without re-running
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF.from_pycapsule">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.WindowUDFExportable" title="datafusion.user_defined.WindowUDFExportable"><spa [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.WindowUDFExportable" title="datafusion.user_defined.WindowUDFExportable"><spa [...]
<dd><p>Create a Window UDF from WindowUDF PyCapsule object.</p>
<p>This function will instantiate a Window UDF that uses a DataFusion
WindowUDF that is exported via the FFI bindings.</p>
@@ -1430,6 +1432,10 @@ WindowUDF that is exported via the FFI bindings.</p>
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
<dt class="sig sig-object py">
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span>< [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.WindowUDFExportable" title="datafusion.user_defined.WindowUDFExportable"><span class="p [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">types.CapsuleType</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">ș [...]
<dd><p>Create a new User-Defined Window Function (UDWF).</p>
<p>This class can be used both as either a function or a decorator.</p>
<dl class="simple">
@@ -1530,7 +1536,7 @@ passed to the constructor (which is ignored on the FFI path).</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.user_defined._is_pycapsule">
-<span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">_is_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">object</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-retur [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">_is_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">object</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-retur [...]
<dd><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> when <code class="docutils literal notranslate"><span class="pre">value</span></code> is a CPython <code class="docutils literal notranslate"><span class="pre">PyCapsule</span></code>.</p>
</dd></dl>
diff --git a/genindex.html b/genindex.html
index 800bf6b1..2d224b27 100644
--- a/genindex.html
+++ b/genindex.html
@@ -1066,6 +1066,8 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Analyze">Analyze (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.any_match">any_match() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.any_value">any_value() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.InsertOp.APPEND">APPEND (datafusion.dataframe.InsertOp attribute)</a>
@@ -1087,6 +1089,8 @@
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.array">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_add">array_add() (in module datafusion.functions)</a>
+</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_agg">array_agg() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_any_match">array_any_match() (in module datafusion.functions)</a>
@@ -1094,6 +1098,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_any_value">array_any_value() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_append">array_append() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_avg">array_avg() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_cat">array_cat() (in module datafusion.functions)</a>
</li>
@@ -1140,6 +1146,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_extract">array_extract() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_filter">array_filter() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_first">array_first() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_has">array_has() (in module datafusion.functions)</a>
</li>
@@ -1153,8 +1161,6 @@
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_join">array_join() (in module datafusion.functions)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.array_length">array_length() (datafusion.Expr method)</a>
<ul>
@@ -1163,6 +1169,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_length">(in module datafusion.functions)</a>
</li>
</ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_max">array_max() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_min">array_min() (in module datafusion.functions)</a>
@@ -1198,6 +1206,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_positions">array_positions() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_prepend">array_prepend() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_product">array_product() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_push_back">array_push_back() (in module datafusion.functions)</a>
</li>
@@ -1224,10 +1234,16 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_resize">array_resize() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_reverse">array_reverse() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_scale">array_scale() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_slice">array_slice() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_sort">array_sort() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_subtract">array_subtract() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_sum">array_sum() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.array_to_string">array_to_string() (in module datafusion.functions)</a>
</li>
@@ -1296,7 +1312,11 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.atan2">atan2() (in module datafusion.functions)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.atan2">(in module datafusion.functions.spark)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.atanh">atanh() (datafusion.Expr method)</a>
<ul>
@@ -1415,6 +1435,8 @@
<li><a href="autoapi/datafusion/input/index.html#datafusion.input.LocationInputPlugin.build_table">(datafusion.input.LocationInputPlugin method)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainMetricCategory.BYTES">BYTES (datafusion.dataframe.ExplainMetricCategory attribute)</a>
+</li>
</ul></td>
</tr></table>
@@ -1511,6 +1533,8 @@
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.ceil">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.ceiling">ceiling() (in module datafusion.functions.spark)</a>
+</li>
<li><a href="autoapi/datafusion/dataframe_formatter/index.html#datafusion.dataframe_formatter.CellFormatter">CellFormatter (class in datafusion.dataframe_formatter)</a>
</li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.char">char() (in module datafusion.functions.spark)</a>
@@ -1521,6 +1545,8 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.char_length">(datafusion.expr.Expr method)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.char_length">(in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.char_length">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr.character_length">character_length() (datafusion.Expr method)</a>
@@ -1529,6 +1555,8 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.character_length">(datafusion.expr.Expr method)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.character_length">(in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.character_length">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.ExecutionPlan.children">children() (datafusion.ExecutionPlan method)</a>
@@ -1634,7 +1662,11 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.concat_ws">concat_ws() (in module datafusion.functions)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.concat_ws">(in module datafusion.functions.spark)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.RuntimeEnvBuilder.config_internal">config_internal (datafusion.context.RuntimeEnvBuilder attribute)</a>
<ul>
@@ -1703,8 +1735,12 @@
</li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.crc32">crc32() (in module datafusion.functions.spark)</a>
</li>
- <li><a href="autoapi/datafusion/index.html#datafusion.TableProviderFactory.create">create() (datafusion.TableProviderFactory method)</a>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.TableProviderFactory.create">create() (datafusion.catalog.TableProviderFactory method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/index.html#datafusion.TableProviderFactory.create">(datafusion.TableProviderFactory method)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.create_dataframe">create_dataframe() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.create_dataframe_from_logical_plan">create_dataframe_from_logical_plan() (datafusion.context.SessionContext method)</a>
@@ -1954,8 +1990,16 @@
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.date_trunc">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.dateadd">dateadd() (in module datafusion.functions.spark)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.datediff">datediff() (in module datafusion.functions.spark)</a>
+</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.datepart">datepart() (in module datafusion.functions)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.datepart">(in module datafusion.functions.spark)</a>
</li>
+ </ul></li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.datetrunc">datetrunc() (in module datafusion.functions)</a>
@@ -2019,6 +2063,8 @@
<li><a href="autoapi/datafusion/substrait/index.html#datafusion.substrait.Serde.deserialize">deserialize() (datafusion.substrait.Serde static method)</a>
</li>
<li><a href="autoapi/datafusion/substrait/index.html#datafusion.substrait.Serde.deserialize_bytes">deserialize_bytes() (datafusion.substrait.Serde static method)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainAnalyzeLevel.DEV">DEV (datafusion.dataframe.ExplainAnalyzeLevel attribute)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.df">df (datafusion.dataframe.DataFrame attribute)</a>
</li>
@@ -2178,10 +2224,10 @@
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowEvaluator.evaluate">(datafusion.user_defined.WindowEvaluator method)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowEvaluator.evaluate_all">evaluate_all() (datafusion.user_defined.WindowEvaluator method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowEvaluator.evaluate_all_with_rank">evaluate_all_with_rank() (datafusion.user_defined.WindowEvaluator method)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.except_all">except_all() (datafusion.dataframe.DataFrame method)</a>
@@ -2217,6 +2263,8 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Explain">Explain (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.explain">explain() (datafusion.dataframe.DataFrame method)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainAnalyzeLevel">ExplainAnalyzeLevel (class in datafusion.dataframe)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.ExplainFormat">ExplainFormat (class in datafusion)</a>
@@ -2224,6 +2272,8 @@
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainFormat">(class in datafusion.dataframe)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainMetricCategory">ExplainMetricCategory (class in datafusion.dataframe)</a>
+</li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.expm1">expm1() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.Expr">Expr (class in datafusion)</a>
@@ -2274,6 +2324,8 @@
<li><a href="autoapi/datafusion/options/index.html#datafusion.options.CsvReadOptions.file_extension">(datafusion.options.CsvReadOptions attribute)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.file_row_index">file_row_index() (in module datafusion.functions)</a>
+</li>
<li><a href="autoapi/datafusion/index.html#datafusion.CsvReadOptions.file_sort_order">file_sort_order (datafusion.CsvReadOptions attribute)</a>
<ul>
@@ -2282,9 +2334,11 @@
</ul></li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.FileType">FileType (in module datafusion.expr)</a>
</li>
- <li><a href="autoapi/datafusion/index.html#datafusion.Expr.fill_nan">fill_nan() (datafusion.Expr method)</a>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.fill_nan">fill_nan() (datafusion.dataframe.DataFrame method)</a>
<ul>
+ <li><a href="autoapi/datafusion/index.html#datafusion.Expr.fill_nan">(datafusion.Expr method)</a>
+</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.fill_nan">(datafusion.expr.Expr method)</a>
</li>
</ul></li>
@@ -2330,10 +2384,10 @@
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.floor">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
- <li><a href="autoapi/datafusion/dataframe_formatter/index.html#datafusion.dataframe_formatter.DataFrameHtmlFormatter.format_html">format_html() (datafusion.dataframe_formatter.DataFrameHtmlFormatter method)</a>
-</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="autoapi/datafusion/dataframe_formatter/index.html#datafusion.dataframe_formatter.DataFrameHtmlFormatter.format_html">format_html() (datafusion.dataframe_formatter.DataFrameHtmlFormatter method)</a>
+</li>
<li><a href="autoapi/datafusion/dataframe_formatter/index.html#datafusion.dataframe_formatter.DataFrameHtmlFormatter.format_str">format_str() (datafusion.dataframe_formatter.DataFrameHtmlFormatter method)</a>
</li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.format_string">format_string() (in module datafusion.functions.spark)</a>
@@ -2467,6 +2521,8 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.get_worker_ctx">get_worker_ctx() (in module datafusion.ipc)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.getbit">getbit() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.global_ctx">global_ctx() (datafusion.context.SessionContext class method)</a>
</li>
@@ -2506,10 +2562,10 @@
<li><a href="autoapi/datafusion/plan/index.html#datafusion.plan.PhysicalPartitioning.hash_expressions">(datafusion.plan.PhysicalPartitioning property)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.head">head() (datafusion.dataframe.DataFrame method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.hex">hex() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.HigherOrderFunction">HigherOrderFunction (in module datafusion.expr)</a>
@@ -2517,6 +2573,8 @@
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.hour">hour() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/object_store/index.html#datafusion.object_store.Http">Http (in module datafusion.object_store)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.hypot">hypot() (in module datafusion.functions.spark)</a>
</li>
</ul></td>
</tr></table>
@@ -2555,6 +2613,8 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.InList">InList (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.inner_product">inner_product() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.input_file_name">input_file_name() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.LogicalPlan.inputs">inputs() (datafusion.LogicalPlan method)</a>
@@ -2735,12 +2795,16 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Limit">Limit (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.limit">limit() (datafusion.dataframe.DataFrame method)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_add">list_add() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_any_match">list_any_match() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_any_value">list_any_value() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_append">list_append() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_avg">list_avg() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_cat">list_cat() (in module datafusion.functions)</a>
</li>
@@ -2777,6 +2841,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_extract">list_extract() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_filter">list_filter() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_first">list_first() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_has">list_has() (in module datafusion.functions)</a>
</li>
@@ -2814,10 +2880,10 @@
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_overlap">list_overlap() (in module datafusion.functions)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_pop_back">list_pop_back() (in module datafusion.functions)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_pop_front">list_pop_front() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_position">list_position() (in module datafusion.functions)</a>
@@ -2825,6 +2891,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_positions">list_positions() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_prepend">list_prepend() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_product">list_product() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_push_back">list_push_back() (in module datafusion.functions)</a>
</li>
@@ -2847,10 +2915,16 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_resize">list_resize() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_reverse">list_reverse() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_scale">list_scale() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_slice">list_slice() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_sort">list_sort() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_subtract">list_subtract() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_sum">list_sum() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.list_to_string">list_to_string() (in module datafusion.functions)</a>
</li>
@@ -3038,10 +3112,10 @@
</ul></li>
<li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema.memory_schema">memory_schema() (datafusion.catalog.Schema static method)</a>
</li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.Accumulator.merge">merge() (datafusion.user_defined.Accumulator method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/index.html#datafusion.Metric">Metric (class in datafusion)</a>
<ul>
@@ -3120,6 +3194,8 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.modulus">modulus() (in module datafusion.functions.spark)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.monthname">monthname() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/unparser/index.html#datafusion.unparser.Dialect.mysql">mysql() (datafusion.unparser.Dialect static method)</a>
</li>
@@ -3329,10 +3405,10 @@
<li><a href="autoapi/datafusion/index.html#datafusion.ExplainFormat.PGJSON">(datafusion.ExplainFormat attribute)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.physical_extension_codec_ids">physical_extension_codec_ids() (datafusion.context.SessionContext method)</a>
</li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/extensions/index.html#datafusion.extensions.SessionExtensionComponents.physical_extension_codecs">physical_extension_codecs (datafusion.extensions.SessionExtensionComponents attribute)</a>
</li>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.PhysicalExtensionCodecExportable">PhysicalExtensionCodecExportable (class in datafusion.user_defined)</a>
@@ -3362,10 +3438,20 @@
<li><a href="autoapi/datafusion/unparser/index.html#datafusion.unparser.Dialect.postgres">postgres() (datafusion.unparser.Dialect static method)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.pow">pow() (in module datafusion.functions)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.pow">(in module datafusion.functions.spark)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.power">power() (in module datafusion.functions)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.power">(in module datafusion.functions.spark)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Prepare">Prepare (in module datafusion.expr)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.printf">printf() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/substrait/index.html#datafusion.substrait.Producer">Producer (class in datafusion.substrait)</a>
</li>
@@ -3395,6 +3481,8 @@
<li><a href="autoapi/datafusion/options/index.html#datafusion.options.CsvReadOptions.quote">(datafusion.options.CsvReadOptions attribute)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.quote">quote() (in module datafusion.functions.spark)</a>
+</li>
</ul></td>
</tr></table>
@@ -3409,6 +3497,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.radians">(in module datafusion.functions)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.rand">rand() (in module datafusion.functions)</a>
+</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.random">random() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.range">range() (in module datafusion.functions)</a>
@@ -3642,6 +3732,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.row">row() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.row_number">row_number() (in module datafusion.functions)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainMetricCategory.ROWS">ROWS (datafusion.dataframe.ExplainMetricCategory attribute)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.rpad">rpad() (in module datafusion.functions)</a>
</li>
@@ -3780,6 +3872,8 @@
<li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.set_worker_ctx">set_worker_ctx() (in module datafusion.ipc)</a>
</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.SetVariable">SetVariable (in module datafusion.expr)</a>
+</li>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.sha">sha() (in module datafusion.functions.spark)</a>
</li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.sha1">sha1() (in module datafusion.functions.spark)</a>
</li>
@@ -3986,7 +4080,11 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.SubqueryAlias">SubqueryAlias (in module datafusion.expr)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.substr">substr() (in module datafusion.functions)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.substr">(in module datafusion.functions.spark)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.substr_index">substr_index() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.substring">substring() (in module datafusion.functions)</a>
@@ -3995,6 +4093,8 @@
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.substring">(in module datafusion.functions.spark)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.substring_index">substring_index() (in module datafusion.functions)</a>
+</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.sum">sum() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.MetricsSet.sum_by_name">sum_by_name() (datafusion.MetricsSet method)</a>
@@ -4003,6 +4103,8 @@
<li><a href="autoapi/datafusion/plan/index.html#datafusion.plan.MetricsSet.sum_by_name">(datafusion.plan.MetricsSet method)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainAnalyzeLevel.SUMMARY">SUMMARY (datafusion.dataframe.ExplainAnalyzeLevel attribute)</a>
+</li>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowEvaluator.supports_bounded_execution">supports_bounded_execution() (datafusion.user_defined.WindowEvaluator method)</a>
</li>
</ul></td>
@@ -4052,7 +4154,11 @@
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.TableProviderExportable">TableProviderExportable (class in datafusion.context)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.TableProviderFactory">TableProviderFactory (class in datafusion)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/catalog/index.html#datafusion.catalog.TableProviderFactory">(class in datafusion.catalog)</a>
</li>
+ </ul></li>
<li><a href="autoapi/datafusion/index.html#datafusion.TableProviderFactoryExportable">TableProviderFactoryExportable (class in datafusion)</a>
</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.TableScan">TableScan (in module datafusion.expr)</a>
@@ -4082,6 +4188,8 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.time_trunc">time_trunc() (in module datafusion.functions.spark)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainMetricCategory.TIMING">TIMING (datafusion.dataframe.ExplainMetricCategory attribute)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.to_arrow_table">to_arrow_table() (datafusion.dataframe.DataFrame method)</a>
</li>
@@ -4122,11 +4230,11 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.to_local_time">to_local_time() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.to_pandas">to_pandas() (datafusion.dataframe.DataFrame method)</a>
-</li>
- <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.to_polars">to_polars() (datafusion.dataframe.DataFrame method)</a>
</li>
</ul></td>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.to_polars">to_polars() (datafusion.dataframe.DataFrame method)</a>
+</li>
<li><a href="autoapi/datafusion/index.html#datafusion.ExecutionPlan.to_proto">to_proto() (datafusion.ExecutionPlan method)</a>
<ul>
@@ -4280,6 +4388,8 @@
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.udwfs">udwfs() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.unbase64">unbase64() (in module datafusion.functions.spark)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.ExplainMetricCategory.UNCATEGORIZED">UNCATEGORIZED (datafusion.dataframe.ExplainMetricCategory attribute)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.Compression.UNCOMPRESSED">UNCOMPRESSED (datafusion.dataframe.Compression attribute)</a>
</li>
@@ -4387,6 +4497,8 @@
<h2 id="W">W</h2>
<table style="width: 100%" class="indextable genindextable"><tr>
<td style="width: 33%; vertical-align: top;"><ul>
+ <li><a href="autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.weekday">weekday() (in module datafusion.functions.spark)</a>
+</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.CaseBuilder.when">when() (datafusion.expr.CaseBuilder method)</a>
<ul>
@@ -4537,14 +4649,14 @@
<li><a href="autoapi/datafusion/options/index.html#datafusion.options.CsvReadOptions.with_file_sort_order">(datafusion.options.CsvReadOptions method)</a>
</li>
</ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.RuntimeEnvBuilder.with_greedy_memory_pool">with_greedy_memory_pool() (datafusion.context.RuntimeEnvBuilder method)</a>
<ul>
<li><a href="autoapi/datafusion/index.html#datafusion.RuntimeEnvBuilder.with_greedy_memory_pool">(datafusion.RuntimeEnvBuilder method)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/index.html#datafusion.CsvReadOptions.with_has_header">with_has_header() (datafusion.CsvReadOptions method)</a>
<ul>
diff --git a/objects.inv b/objects.inv
index 3ef6b03d..3d970161 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/searchindex.js b/searchindex.js
index 3c62cc64..8be98dff 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"A bundle you can run": [[26, "a-bundle-you-can-run"]], "A codec decodes against the session running the query": [[29, "a-codec-decodes-against-the-session-running-the-query"]], "A physical optimizer rule": [[32, "a-physical-optimizer-rule"]], "A table provider": [[35, "a-table-provider"]], "A table provider factory": [[35, "a-table-provider-factory"]], "API Reference": [[21, null], [54, "api-reference"]], "Accessing the Calling Session": [[49, "accessing-t [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"A bundle you can run": [[26, "a-bundle-you-can-run"]], "A codec decodes against the session running the query": [[29, "a-codec-decodes-against-the-session-running-the-query"]], "A physical optimizer rule": [[32, "a-physical-optimizer-rule"]], "A table provider": [[35, "a-table-provider"]], "A table provider factory": [[35, "a-table-provider-factory"]], "API Reference": [[21, null], [54, "api-reference"]], "Accessing the Calling Session": [[49, "accessing-t [...]
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html b/user-guide/common-operations/aggregations.html
index cb4f4281..370fd5aa 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -1252,7 +1252,7 @@ Where <code class="docutils literal notranslate"><span class="pre">grouping(Type
<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.regr_avgy" title="datafusion.functions.regr_avgy"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.regr_avgy()</span></code></a></p></li>
<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.regr_sxx" title="datafusion.functions.regr_sxx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.regr_sxx()</span></code></a></p></li>
<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.regr_syy" title="datafusion.functions.regr_syy"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.regr_syy()</span></code></a></p></li>
-<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.regr_slope" title="datafusion.functions.regr_slope"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.regr_slope()</span></code></a></p></li>
+<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.regr_sxy" title="datafusion.functions.regr_sxy"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.regr_sxy()</span></code></a></p></li>
</ul>
</dd>
</dl>
@@ -1262,6 +1262,7 @@ Where <code class="docutils literal notranslate"><span class="pre">grouping(Type
<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.first_value" title="datafusion.functions.first_value"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.first_value()</span></code></a></p></li>
<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.last_value" title="datafusion.functions.last_value"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.last_value()</span></code></a></p></li>
<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.nth_value" title="datafusion.functions.nth_value"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.nth_value()</span></code></a></p></li>
+<li><p><a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.any_value" title="datafusion.functions.any_value"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.any_value()</span></code></a></p></li>
</ul>
</dd>
</dl>
diff --git a/user-guide/common-operations/basic-info.html b/user-guide/common-operations/basic-info.html
index 7f957b61..844bf898 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -565,11 +565,11 @@
+-----+---------+--------+--------+
| nrs | names | random | groups |
+-----+---------+--------+--------+
-| 1 | python | 154 | A |
-| 2 | ruby | 979 | A |
-| 3 | java | 342 | B |
-| 4 | haskell | 11 | C |
-| 5 | go | 816 | B |
+| 1 | python | 600 | A |
+| 2 | ruby | 671 | A |
+| 3 | java | 235 | B |
+| 4 | haskell | 673 | C |
+| 5 | go | 410 | B |
+-----+---------+--------+--------+
</pre></div>
</div>
@@ -587,8 +587,8 @@
+-----+--------+--------+--------+
| nrs | names | random | groups |
+-----+--------+--------+--------+
-| 1 | python | 154 | A |
-| 2 | ruby | 979 | A |
+| 1 | python | 600 | A |
+| 2 | ruby | 671 | A |
+-----+--------+--------+--------+
</pre></div>
</div>
@@ -620,11 +620,11 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> nrs names random groups
-0 1 python 154 A
-1 2 ruby 979 A
-2 3 java 342 B
-3 4 haskell 11 C
-4 5 go 816 B
+0 1 python 600 A
+1 2 ruby 671 A
+2 3 java 235 B
+3 4 haskell 673 C
+4 5 go 410 B
</pre></div>
</div>
</div>
@@ -638,17 +638,17 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>DataFrame()
-+------------+--------------------+-------+--------------------+--------+
-| describe | nrs | names | random | groups |
-+------------+--------------------+-------+--------------------+--------+
-| count | 5.0 | 5 | 5.0 | 5 |
-| null_count | 0.0 | 0 | 0.0 | 0 |
-| mean | 3.0 | null | 460.4 | null |
-| std | 1.5811388300841898 | null | 419.89796379596794 | null |
-| min | 1.0 | go | 11.0 | A |
-| max | 5.0 | ruby | 979.0 | C |
-| median | 3.0 | null | 342.0 | null |
-+------------+--------------------+-------+--------------------+--------+
++------------+--------------------+-------+-------------------+--------+
+| describe | nrs | names | random | groups |
++------------+--------------------+-------+-------------------+--------+
+| count | 5.0 | 5 | 5.0 | 5 |
+| null_count | 0.0 | 0 | 0.0 | 0 |
+| mean | 3.0 | null | 517.8 | null |
+| std | 1.5811388300841898 | null | 190.9913610611747 | null |
+| min | 1.0 | go | 235.0 | A |
+| max | 5.0 | ruby | 673.0 | C |
+| median | 3.0 | null | 600.0 | null |
++------------+--------------------+-------+-------------------+--------+
</pre></div>
</div>
</div>
diff --git a/user-guide/common-operations/expressions.html b/user-guide/common-operations/expressions.html
index 015f4608..3607baf5 100644
--- a/user-guide/common-operations/expressions.html
+++ b/user-guide/common-operations/expressions.html
@@ -751,7 +751,9 @@ This function returns a new array with the elements repeated.</p>
<p>Some array functions take a <em>lambda function</em>: a small function that runs once
per element. <a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.array_transform" title="datafusion.functions.array_transform"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_transform()</span></code></a> maps a lambda over
every element, <a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.array_filter" title="datafusion.functions.array_filter"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_filter()</span></code></a> keeps the elements
-for which a predicate lambda is true, and
+for which a predicate lambda is true,
+<a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.array_first" title="datafusion.functions.array_first"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_first()</span></code></a> returns the first element that
+satisfies a predicate lambda, and
<a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.array_any_match" title="datafusion.functions.array_any_match"><code class="xref py py-func docutils literal notranslate"><span class="pre">array_any_match()</span></code></a> returns whether any element
satisfies a predicate lambda. (Functions that take another function as an
argument are sometimes called <em>higher-order</em> functions.)</p>
@@ -766,6 +768,7 @@ become the lambda parameters, and its return value becomes the body.</p>
<span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"a"</span><span class="p">:</span> <span class="p">[[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="mi">5</ [...]
<span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">array_transform</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="k">lambda</span> <span class="n">v</span><span class="p">:</span> <span class="n">v</span> <span class="o">*</span> <span class="mi">2</span><span class=" [...]
<span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">array_filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="k">lambda</span> <span class="n">v</span><span class="p">:</span> <span class="n">v</span> <span class="o">></span> <span class="mi">2</span><span class=" [...]
+<span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">array_first</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="k">lambda</span> <span class="n">v</span><span class="p">:</span> <span class="n">v</span> <span class="o">></span> <span class="mi">2</span><span class="p [...]
<span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">array_any_match</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="k">lambda</span> <span class="n">v</span><span class="p">:</span> <span class="n">v</span> <span class="o">></span> <span class="mi">3</span><span clas [...]
</pre></div>
</div>
diff --git a/user-guide/common-operations/functions.html b/user-guide/common-operations/functions.html
index 3d5ac2e6..397f828a 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -648,16 +648,16 @@ NULL-propagation, 1-indexed substrings, HALF_UP rounding, etc.), see
+-------------------------------+
| now() |
+-------------------------------+
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
-| 2026-09-29T18:58:48.954962516 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+| 2026-10-04T16:25:56.059570654 |
+-------------------------------+
Data truncated.
</pre></div>
@@ -882,6 +882,35 @@ Data truncated.
</div>
<p>The fill value will be cast to match each column’s type. If casting fails for a column, that column remains unchanged.</p>
</section>
+<section id="fill-nan">
+<h2>fill_nan<a class="headerlink" href="#fill-nan" title="Link to this heading">#</a></h2>
+<p>The <code class="docutils literal notranslate"><span class="pre">fill_nan()</span></code> method replaces NaN values in floating-point columns. NaN is
+distinct from NULL, which <code class="docutils literal notranslate"><span class="pre">fill_null()</span></code> handles:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># Replace NaN with 0.0 in every floating-point column</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">)</span>
+
+<span class="c1"># Replace NaN only in specific columns</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">subset</span><span class="o">=</span><span class="p">[</span><span class="s2">"price"</span><span class="p">])</span>
+</pre></div>
+</div>
+</section>
+<section id="column-names-with-uppercase-letters-or-dots">
+<span id="fill-column-names"></span><h2>Column names with uppercase letters or dots<a class="headerlink" href="#column-names-with-uppercase-letters-or-dots" title="Link to this heading">#</a></h2>
+<p><code class="docutils literal notranslate"><span class="pre">fill_null()</span></code> and <code class="docutils literal notranslate"><span class="pre">fill_nan()</span></code> fail on a DataFrame that has any column whose
+name contains an uppercase letter or a <code class="docutils literal notranslate"><span class="pre">.</span></code>, even when that column is not in
+<code class="docutils literal notranslate"><span class="pre">subset</span></code>
+(<a class="reference external" href="https://github.com/apache/datafusion/issues/25829">apache/datafusion#25829</a>):</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">({</span><span class="s2">"Price"</span><span class="p">:</span> <span class="p">[</span><span class="nb">float</span><span class="p">(</span><span class="s2">"nan"</span><span class="p">),</span> <span class="mf">2.0</span><span class="p [...]
+<span class="n">df</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">subset</span><span class="o">=</span><span class="p">[</span><span class="s2">"qty"</span><span class="p">])</span>
+<span class="c1"># Schema error: No field named price. Did you mean '..."Price"'?</span>
+</pre></div>
+</div>
+<p>Until that is fixed, rename such columns first. Quote the old name so it is
+not normalized:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">df</span><span class="o">.</span><span class="n">with_column_renamed</span><span class="p">(</span><span class="s1">'"Price"'</span><span class="p">,</span> <span class="s2">"price"</span><span class="p">)</span><span class="o">.</span><span class="n">fill_nan</span><span class="p">(</span><span class="mf">0.0</span><span class="p">,</span> <span class="n">subset [...]
+</pre></div>
+</div>
+</section>
</section>
@@ -942,6 +971,8 @@ Data truncated.
</li>
<li class="toc-h1 nav-item toc-entry"><a class="reference internal nav-link" href="#handling-missing-values">Handling Missing Values</a><ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fill-null">fill_null</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#fill-nan">fill_nan</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#column-names-with-uppercase-letters-or-dots">Column names with uppercase letters or dots</a></li>
</ul>
</li>
</ul>
diff --git a/user-guide/common-operations/index.html b/user-guide/common-operations/index.html
index 9ea0420c..ea8c338d 100644
--- a/user-guide/common-operations/index.html
+++ b/user-guide/common-operations/index.html
@@ -579,6 +579,8 @@
</li>
<li class="toctree-l1"><a class="reference internal" href="functions.html#handling-missing-values">Handling Missing Values</a><ul>
<li class="toctree-l2"><a class="reference internal" href="functions.html#fill-null">fill_null</a></li>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#fill-nan">fill_nan</a></li>
+<li class="toctree-l2"><a class="reference internal" href="functions.html#column-names-with-uppercase-letters-or-dots">Column names with uppercase letters or dots</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="spark-functions.html">Spark-Compatible Functions</a><ul>
diff --git a/user-guide/common-operations/udf-and-udfa.html b/user-guide/common-operations/udf-and-udfa.html
index da9e1153..9e7e377e 100644
--- a/user-guide/common-operations/udf-and-udfa.html
+++ b/user-guide/common-operations/udf-and-udfa.html
@@ -712,7 +712,7 @@ over column references and literals, so the optimizer can analyze it:</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>FilterExec: brand@1 = A AND qty@2 >= 150
RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
- DataSourceExec: file_groups={1 group: [[tmp/tmp3tz5tm4x/items.parquet]]}, projection=[id, brand, qty], file_type=parquet, predicate=brand@1 = A AND qty@2 >= 150, pruning_predicate=brand_null_count@2 != row_count@3 AND brand_min@0 <= A AND A <= brand_max@1 AND qty_null_count@5 != row_count@3 AND qty_max@4 >= 150, required_guarantees=[brand in (A)]
+ DataSourceExec: file_groups={1 group: [[tmp/tmpu0a3nnts/items.parquet]]}, projection=[id, brand, qty], file_type=parquet, predicate=brand@1 = A AND qty@2 >= 150, pruning_predicate=brand_null_count@2 != row_count@3 AND brand_min@0 <= A AND A <= brand_max@1 AND qty_null_count@5 != row_count@3 AND qty_max@4 >= 150, required_guarantees=[brand in (A)]
</pre></div>
</div>
</div>
@@ -748,7 +748,7 @@ bloom filter or dictionary is available to skip pages.</p></li>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>FilterExec: brand_qty_filter(CAST(brand@1 AS Utf8), qty@2)
RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
- DataSourceExec: file_groups={1 group: [[tmp/tmp3tz5tm4x/items.parquet]]}, projection=[id, brand, qty], file_type=parquet, predicate=brand_qty_filter(CAST(brand@1 AS Utf8), qty@2)
+ DataSourceExec: file_groups={1 group: [[tmp/tmpu0a3nnts/items.parquet]]}, projection=[id, brand, qty], file_type=parquet, predicate=brand_qty_filter(CAST(brand@1 AS Utf8), qty@2)
</pre></div>
</div>
</div>
diff --git a/user-guide/common-operations/windows.html b/user-guide/common-operations/windows.html
index f97d1289..af94e7c2 100644
--- a/user-guide/common-operations/windows.html
+++ b/user-guide/common-operations/windows.html
@@ -741,12 +741,47 @@ Data truncated.
</div>
</div>
</div>
+<section id="chaining-onto-a-window-function">
+<span id="window-function-chaining"></span><h4>Chaining onto a window function<a class="headerlink" href="#chaining-onto-a-window-function" title="Link to this heading">#</a></h4>
+<p>Set a window function’s <code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, and <code class="docutils literal notranslate"><span class="pre">window_frame</span></code> in one
+place: its keyword arguments, a single <code class="docutils literal notranslate"><span class="pre">over()</span></code>, or one builder chain ending in
+<code class="docutils literal notranslate"><span class="pre">build()</span></code>. Chaining a builder method or <code class="docutils literal notranslate"><span class="pre">over()</span></code> onto a window function that
+already has any of them raises:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># Raises: lead already has window options (order_by)</span>
+<span class="n">f</span><span class="o">.</span><span class="n">lead</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="n">order_by</span><span class="o">=</span><span class="s2">"t"</span><span class="p">)</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">(</span><span class="n">partition_by</span>< [...]
+
+<span class="c1"># Set them together instead.</span>
+<span class="n">f</span><span class="o">.</span><span class="n">lead</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">(</span><span class="n">partition_by</span><span class="o">=</span><span class="p">[</span><span class="n">col</span><span class="p">(</span><span class="s2">&qu [...]
+</pre></div>
+</div>
+<p>A built window function stores a concrete frame with no record of whether you
+chose it or it was derived from <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, so the options cannot be merged
+without guessing. Merging may become possible once
+<a class="reference external" href="https://github.com/apache/datafusion/issues/25934">apache/datafusion#25934</a>
+is resolved.</p>
+<p>The <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code> already set is kept, as are <code class="docutils literal notranslate"><span class="pre">filter</span></code> and <code class="docutils literal notranslate"><span class="pre">distinct</span></code> on an
+aggregate used as a window function. The whole-partition frame counts as no
+frame, so adding an <code class="docutils literal notranslate"><span class="pre">order_by</span></code> derives the running frame, even when you passed
+that frame explicitly:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">whole</span> <span class="o">=</span> <span class="n">WindowFrame</span><span class="p">(</span><span class="s2">"rows"</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> <span class="c1"># same as the no-order_by default</span>
+
+<span class="c1"># Both give a running sum.</span>
+<span class="n">f</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">())</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">" [...]
+<span class="n">f</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">(</span><span class="n">window_frame</span><span class="o">=</span><span class="n">whole</span><span class="p">))</span><span class="o">.</span><span class="n">or [...]
+</pre></div>
+</div>
+<p>To keep that frame, set it after the <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, or pass both in one <code class="docutils literal notranslate"><span class="pre">Window</span></code>:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">())</span><span class="o">.</span><span class="n">order_by</span><span class="p [...]
+<span class="n">f</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">(</span><span class="n">order_by</span><span class="o">=</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class= [...]
+</pre></div>
+</div>
+</section>
</section>
<section id="null-treatment">
<h3>Null Treatment<a class="headerlink" href="#null-treatment" title="Link to this heading">#</a></h3>
<p>When using aggregate functions as window functions, it is often useful to specify how null values
-should be treated. In order to do this you need to use the builder function. In future releases
-we expect this to be simplified in the interface.</p>
+should be treated. Pass <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code> in the <code class="docutils literal notranslate"><span class="pre">Window</span></code>, or set it on the aggregate itself, which
+<code class="docutils literal notranslate"><span class="pre">over()</span></code> keeps (see <a class="reference internal" href="#aggregate-over-options"><span class="std std-ref">Options set on the aggregate</span></a>).</p>
<p>One common usage for handling nulls is the case where you want to find the last value up to the
current row. In the following example we demonstrate how setting the null treatment to ignore
nulls will fill in with the value of the most recent non-null row. To do this, we also will set
@@ -847,6 +882,25 @@ Data truncated.
</div>
</div>
</div>
+<section id="options-set-on-the-aggregate">
+<span id="aggregate-over-options"></span><h3>Options set on the aggregate<a class="headerlink" href="#options-set-on-the-aggregate" title="Link to this heading">#</a></h3>
+<p><code class="docutils literal notranslate"><span class="pre">over()</span></code> keeps the <code class="docutils literal notranslate"><span class="pre">filter</span></code>, <code class="docutils literal notranslate"><span class="pre">distinct</span></code>, and <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code> options an
+aggregate was built with:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># Averages the distinct values 1.0 and 4.0.</span>
+<span class="n">f</span><span class="o">.</span><span class="n">avg</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="n">distinct</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">())</span>
+</pre></div>
+</div>
+<p>An aggregate’s <code class="docutils literal notranslate"><span class="pre">order_by</span></code> raises, as <code class="docutils literal notranslate"><span class="pre">ORDER</span> <span class="pre">BY</span></code> inside an aggregate call does
+with <code class="docutils literal notranslate"><span class="pre">OVER</span></code> in SQL. A window does not pass an ordering to the aggregate, so
+the <code class="docutils literal notranslate"><span class="pre">order_by</span></code> in the <code class="docutils literal notranslate"><span class="pre">Window</span></code> only sets the frame and the order of rows. The
+one exception is a <code class="docutils literal notranslate"><span class="pre">WITHIN</span> <span class="pre">GROUP</span></code> function such as
+<a class="reference internal" href="../../autoapi/datafusion/functions/index.html#datafusion.functions.percentile_cont" title="datafusion.functions.percentile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">percentile_cont()</span></code></a>, which computes ascending as a
+window: an ascending <code class="docutils literal notranslate"><span class="pre">sort_expression</span></code> is accepted, and a descending one raises.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">percentile_cont</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="mf">0.25</span><span class="p">)</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">())</span> <s [...]
+<span class="n">f</span><span class="o">.</span><span class="n">percentile_cont</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">ascending</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <span class="mf">0.25</span><span class="p">)</span><span class="o">.</span><span cla [...]
+</pre></div>
+</div>
+</section>
</section>
<section id="available-functions">
<h2>Available Functions<a class="headerlink" href="#available-functions" title="Link to this heading">#</a></h2>
@@ -951,11 +1005,17 @@ must be importable on the receiving worker. See
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#setting-parameters">Setting Parameters</a><ul class="visible nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#ordering">Ordering</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#partitions">Partitions</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#window-frame">Window Frame</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#window-frame">Window Frame</a><ul class="nav section-nav flex-column">
+<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#chaining-onto-a-window-function">Chaining onto a window function</a></li>
+</ul>
+</li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#null-treatment">Null Treatment</a></li>
</ul>
</li>
-<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#aggregate-functions">Aggregate Functions</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#aggregate-functions">Aggregate Functions</a><ul class="visible nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#options-set-on-the-aggregate">Options set on the aggregate</a></li>
+</ul>
+</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#available-functions">Available Functions</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#user-defined-window-functions">User-Defined Window Functions</a></li>
</ul>
diff --git a/user-guide/upgrade-guides.html b/user-guide/upgrade-guides.html
index 8c8ec90d..480e1fae 100644
--- a/user-guide/upgrade-guides.html
+++ b/user-guide/upgrade-guides.html
@@ -700,6 +700,155 @@ any call passing it by keyword.</p>
</pre></div>
</div>
</section>
+<section id="more-aggregate-functions-accept-distinct">
+<h3>More aggregate functions accept <code class="docutils literal notranslate"><span class="pre">distinct</span></code><a class="headerlink" href="#more-aggregate-functions-accept-distinct" title="Link to this heading">#</a></h3>
+<p><a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.bit_and" title="datafusion.functions.bit_and"><code class="xref py py-func docutils literal notranslate"><span class="pre">bit_and()</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.bit_or" title="datafusion.functions.bit_or"><code class="xref py py-func docutils literal notranslate"><span class="pre">bit_or()</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.mean" title="datafusion.functions.mean"><code class="xref py py-func docutils literal notranslate"><span class="pre">mean()</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.percentile_cont" title="datafusion.functions.percentile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">percentile_cont()</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.quantile_cont" title="datafusion.functions.quantile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">quantile_cont()</span></code></a>, and
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.string_agg" title="datafusion.functions.string_agg"><code class="xref py py-func docutils literal notranslate"><span class="pre">string_agg()</span></code></a> now accept a <code class="docutils literal notranslate"><span class="pre">distinct</span></code> argument.
+As with <code class="docutils literal notranslate"><span class="pre">sum</span></code> and <code class="docutils literal notranslate"><span class="pre">avg</span></code> in 54.0.0, <code class="docutils literal notranslate"><span class="pre">distinct</span></code> is inserted <em>before</em> <code class="docutils literal notranslate"><span class="pre">filter</span></code>, so
+code that passed <code class="docutils literal notranslate"><span class="pre">filter</span></code> (or, for <code class="docutils literal notranslate"><span class="pre">string_agg</span></code>, <code class="docutils literal notranslate"><span class="pre">order_by</span></code>) positionally must
+pass it by keyword.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">bit_and</span><span class="p">(</span><span class="n">column</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="n">my_filter</span><span class="p">)</span> <span class="c1"># before</span>
+<span class="n">f</span><span class="o">.</span><span class="n">bit_and</span><span class="p">(</span><span class="n">column</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="nb">filter</span><span class="o">=</span><span class="n">my_filter</span><span class="p">)</span> <span class="c1"># after</span>
+</pre></div>
+</div>
+<p>Passing <code class="docutils literal notranslate"><span class="pre">filter</span></code> to <code class="docutils literal notranslate"><span class="pre">mean</span></code> previously raised a <code class="docutils literal notranslate"><span class="pre">TypeError</span></code>, whether passed
+positionally or by keyword; it now works when passed by keyword.</p>
+</section>
+<section id="chaining-keeps-options-already-set">
+<h3>Chaining keeps options already set<a class="headerlink" href="#chaining-keeps-options-already-set" title="Link to this heading">#</a></h3>
+<p>Chaining a builder method (<code class="docutils literal notranslate"><span class="pre">order_by</span></code>, <code class="docutils literal notranslate"><span class="pre">filter</span></code>, <code class="docutils literal notranslate"><span class="pre">distinct</span></code>, <code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>,
+<code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, <code class="docutils literal notranslate"><span class="pre">window_frame</span></code>) or <code class="docutils literal notranslate"><span class="pre">over()</span></code> onto a function used to start from
+an empty builder, so options set by the function’s keyword arguments were
+silently reset. On an aggregate they are now kept, which can change results:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">e</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">string_agg</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"s"</span><span class="p">),</span> <span class="s2">","</span><span class="p">,</span> <span class="n">order_by</span><span class="o">=</span><span class="s2">"s [...]
+<span class="n">e</span><span class="o">.</span><span class="n">distinct</span><span class="p">()</span><span class="o">.</span><span class="n">build</span><span class="p">()</span> <span class="c1"># before: order_by dropped; after: kept</span>
+</pre></div>
+</div>
+<p>On a window function that already has a <code class="docutils literal notranslate"><span class="pre">partition_by</span></code>, <code class="docutils literal notranslate"><span class="pre">order_by</span></code>, or
+<code class="docutils literal notranslate"><span class="pre">window_frame</span></code>, chaining now raises instead of dropping them. Set them in one
+place, as described in <a class="reference internal" href="common-operations/windows.html#window-function-chaining"><span class="std std-ref">Chaining onto a window function</span></a>:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">e</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">lead</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="n">partition_by</span><span class="o">=</span><span class="p">[</span><span class="n [...]
+<span class="n">e</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">(</span><span class="n">order_by</span><span class="o">=</span><span class="s2">"t"</span><span class="p">))</span> <span class="c1"># before: partition dropped; after: raises</span>
+<span class="n">f</span><span class="o">.</span><span class="n">lead</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">(</span><span class="n">partition_by</span><span class="o">=</span><span class="p">[</span><span class="n">col< [...]
+</pre></div>
+</div>
+<p>Options that used to be dropped now take effect, so a chain that ran before
+may now raise. For example, DISTINCT requires the ORDER BY expressions to be
+among the arguments:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">array_agg</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"s"</span><span class="p">),</span> <span class="n">distinct</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span [...]
+<span class="c1"># before: ran without DISTINCT</span>
+<span class="c1"># after: Execution error: In an aggregate with DISTINCT, ORDER BY expressions</span>
+<span class="c1"># must appear in argument list</span>
+</pre></div>
+</div>
+<p>Drop <code class="docutils literal notranslate"><span class="pre">distinct</span></code>, or order by the aggregated column, to get either of the
+results the chain can actually produce.</p>
+<p>An option that does not apply to the function now raises as soon as it is
+set, anywhere in the chain. <code class="docutils literal notranslate"><span class="pre">filter</span></code> and <code class="docutils literal notranslate"><span class="pre">distinct</span></code> need an aggregate,
+including one used as a window function, and <code class="docutils literal notranslate"><span class="pre">partition_by</span></code> and
+<code class="docutils literal notranslate"><span class="pre">window_frame</span></code> need a window function. Later in a chain these were silently
+dropped:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">)</span> <span clas [...]
+<span class="c1"># before: partition_by dropped; after: raises</span>
+</pre></div>
+</div>
+<p>The default <code class="docutils literal notranslate"><span class="pre">RESPECT</span> <span class="pre">NULLS</span></code> set by <code class="docutils literal notranslate"><span class="pre">first_value</span></code>, <code class="docutils literal notranslate"><span class="pre">last_value</span></code>, and <code class="docutils literal notranslate"><span class="pre">nth_value</span></code>
+is also kept, so their generated column names change:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">first_value</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">))</span><span class="o">.</span><span class="n">order_by</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"b"</span><span class="p">))</span> [...]
+<span class="c1"># before: first_value(a) ORDER BY [b ASC NULLS FIRST]</span>
+<span class="c1"># after: first_value(a) RESPECT NULLS ORDER BY [b ASC NULLS FIRST]</span>
+</pre></div>
+</div>
+<p>This now matches the name from <code class="docutils literal notranslate"><span class="pre">f.first_value(col("a"),</span> <span class="pre">order_by=col("b"))</span></code>.
+Code that selects the result by its generated name should <code class="docutils literal notranslate"><span class="pre">alias()</span></code> it instead.</p>
+</section>
+<section id="over-keeps-options-set-on-an-aggregate">
+<h3><code class="docutils literal notranslate"><span class="pre">over()</span></code> keeps options set on an aggregate<a class="headerlink" href="#over-keeps-options-set-on-an-aggregate" title="Link to this heading">#</a></h3>
+<p><code class="docutils literal notranslate"><span class="pre">Expr.over()</span></code> on an aggregate used to drop the <code class="docutils literal notranslate"><span class="pre">filter</span></code>, <code class="docutils literal notranslate"><span class="pre">distinct</span></code>,
+<code class="docutils literal notranslate"><span class="pre">null_treatment</span></code>, and <code class="docutils literal notranslate"><span class="pre">order_by</span></code> options it was built with. The first three are
+now kept, which can change results:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">avg</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="n">distinct</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n"> [...]
+<span class="c1"># v = [1, 1, 4]; before: 2.0, after: 2.5</span>
+</pre></div>
+</div>
+<p>An <code class="docutils literal notranslate"><span class="pre">order_by</span></code> on the aggregate now raises instead of being dropped, as it does
+with <code class="docutils literal notranslate"><span class="pre">OVER</span></code> in SQL. Remove it, or move it into the <code class="docutils literal notranslate"><span class="pre">Window</span></code> if it was meant to
+order the rows:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># before: order_by dropped; after: raises</span>
+<span class="n">f</span><span class="o">.</span><span class="n">first_value</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">),</span> <span class="n">order_by</span><span class="o">=</span><span class="n">col</span><span class="p">(</span><span class="s2">"i"</span><span class="p">)</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">ascending</span> [...]
+ <span class="n">Window</span><span class="p">(</span><span class="n">partition_by</span><span class="o">=</span><span class="p">[</span><span class="n">col</span><span class="p">(</span><span class="s2">"g"</span><span class="p">)])</span>
+<span class="p">)</span>
+
+<span class="c1"># after: the Window orders the rows the aggregate sees</span>
+<span class="n">f</span><span class="o">.</span><span class="n">first_value</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span>
+ <span class="n">Window</span><span class="p">(</span><span class="n">partition_by</span><span class="o">=</span><span class="p">[</span><span class="n">col</span><span class="p">(</span><span class="s2">"g"</span><span class="p">)],</span> <span class="n">order_by</span><span class="o">=</span><span class="p">[</span><span class="n">col</span><span class="p">(</span><span class="s2">"i"</span><span class="p">)</span><span class="o">.</span><span class="n">sort</sp [...]
+<span class="p">)</span>
+</pre></div>
+</div>
+<p>A <code class="docutils literal notranslate"><span class="pre">WITHIN</span> <span class="pre">GROUP</span></code> function such as <code class="docutils literal notranslate"><span class="pre">percentile_cont</span></code> still accepts
+an ascending <code class="docutils literal notranslate"><span class="pre">sort_expression</span></code>, and raises on a descending one, which used to
+give the ascending result. See <a class="reference internal" href="common-operations/windows.html#aggregate-over-options"><span class="std std-ref">Options set on the aggregate</span></a>.</p>
+</section>
+<section id="python-aggregate-udfs-reject-distinct">
+<h3>Python aggregate UDFs reject <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code><a class="headerlink" href="#python-aggregate-udfs-reject-distinct" title="Link to this heading">#</a></h3>
+<p>A Python <a class="reference internal" href="../autoapi/datafusion/user_defined/index.html#datafusion.user_defined.Accumulator" title="datafusion.user_defined.Accumulator"><code class="xref py py-class docutils literal notranslate"><span class="pre">Accumulator</span></code></a> cannot deduplicate its
+input, so a Python aggregate UDF with <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code> counted every row. It now
+raises instead of returning that result:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">my_sum</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"v"</span><span class="p">))</span><span class="o">.</span><span class="n">over</span><span class="p">(</span><span class="n">Window</span><span class="p">())</span><span class="o">.</span><span class="n">distinct</span><span class="p">()</span><span class="o">.</span><span clas [...]
+<span class="c1"># v = [1, 1, 1, 5]; before: 8.0; after: DISTINCT is not supported ...</span>
+</pre></div>
+</div>
+<p>When the optimizer rewrites the query to group by the distinct values first,
+as it does for SQL’s <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">my_sum(DISTINCT</span> <span class="pre">v)</span> <span class="pre">FROM</span> <span class="pre">t</span></code>, the query still runs
+and gives the distinct result.</p>
+</section>
+<section id="percentile-functions-keep-the-sort-direction">
+<h3>Percentile functions keep the sort direction<a class="headerlink" href="#percentile-functions-keep-the-sort-direction" title="Link to this heading">#</a></h3>
+<p><a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.percentile_cont" title="datafusion.functions.percentile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">percentile_cont()</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.quantile_cont" title="datafusion.functions.quantile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">quantile_cont()</span></code></a>,
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.approx_percentile_cont" title="datafusion.functions.approx_percentile_cont"><code class="xref py py-func docutils literal notranslate"><span class="pre">approx_percentile_cont()</span></code></a>, and
+<a class="reference internal" href="../autoapi/datafusion/functions/index.html#datafusion.functions.approx_percentile_cont_with_weight" title="datafusion.functions.approx_percentile_cont_with_weight"><code class="xref py py-func docutils literal notranslate"><span class="pre">approx_percentile_cont_with_weight()</span></code></a> ignored the
+direction of <code class="docutils literal notranslate"><span class="pre">sort_expression</span></code>, so a descending sort gave the ascending result.
+Used as aggregates, they now match <code class="docutils literal notranslate"><span class="pre">WITHIN</span> <span class="pre">GROUP</span> <span class="pre">(ORDER</span> <span class="pre">BY</span> <span class="pre">...</span> <span class="pre">DESC)</span></code> in SQL
+(see <a class="reference internal" href="common-operations/windows.html#aggregate-over-options"><span class="std std-ref">Options set on the aggregate</span></a> for their use in a window):</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">percentile_cont</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span><span class="o">.</span><span class="n">sort</span><span class="p">(</span><span class="n">ascending</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span> <s [...]
+<span class="c1"># a = [1, 2, 3, 4, 5]; before: 2.0, after: 4.0</span>
+</pre></div>
+</div>
+<p>Their generated column names now include the ordering, in the same form as SQL.
+Code that selects the result by its generated name should <code class="docutils literal notranslate"><span class="pre">alias()</span></code> it instead.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">f</span><span class="o">.</span><span class="n">percentile_cont</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">),</span> <span class="mf">0.25</span><span class="p">)</span>
+<span class="c1"># before: percentile_cont(t.a,Float64(0.25))</span>
+<span class="c1"># after: percentile_cont(Float64(0.25)) WITHIN GROUP [t.a ASC NULLS FIRST]</span>
+</pre></div>
+</div>
+</section>
+<section id="fill-null-subset-fills-no-columns">
+<h3><code class="docutils literal notranslate"><span class="pre">fill_null(subset=[])</span></code> fills no columns<a class="headerlink" href="#fill-null-subset-fills-no-columns" title="Link to this heading">#</a></h3>
+<p><a class="reference internal" href="../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.fill_null" title="datafusion.dataframe.DataFrame.fill_null"><code class="xref py py-meth docutils literal notranslate"><span class="pre">fill_null()</span></code></a> with an empty <code class="docutils literal notranslate"><span class="pre">subset</span></code>
+list used to fill every column, the same as <code class="docutils literal notranslate"><span class="pre">subset=None</span></code>. It now returns the
+DataFrame unchanged, so a subset computed from the schema that matches nothing
+no longer rewrites every column. The new
+<a class="reference internal" href="../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.fill_nan" title="datafusion.dataframe.DataFrame.fill_nan"><code class="xref py py-meth docutils literal notranslate"><span class="pre">fill_nan()</span></code></a> behaves the same way.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">df</span><span class="o">.</span><span class="n">fill_null</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">subset</span><span class="o">=</span><span class="p">[])</span> <span class="c1"># before: fills all columns; after: fills none</span>
+<span class="n">df</span><span class="o">.</span><span class="n">fill_null</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># fills all columns, before and after</span>
+</pre></div>
+</div>
+</section>
+<section id="spark-last-day-renamed-its-parameter">
+<h3><code class="docutils literal notranslate"><span class="pre">spark.last_day</span></code> renamed its parameter<a class="headerlink" href="#spark-last-day-renamed-its-parameter" title="Link to this heading">#</a></h3>
+<p>The parameter of <a class="reference internal" href="../autoapi/datafusion/functions/spark/index.html#datafusion.functions.spark.last_day" title="datafusion.functions.spark.last_day"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.functions.spark.last_day()</span></code></a> is now named
+<code class="docutils literal notranslate"><span class="pre">date</span></code>, matching <code class="docutils literal notranslate"><span class="pre">pyspark.sql.functions.last_day</span></code>. Positional calls are
+unaffected; update any call passing it by keyword.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">spark</span><span class="o">.</span><span class="n">last_day</span><span class="p">(</span><span class="n">col</span><span class="o">=</span><span class="n">d</span><span class="p">)</span> <span class="c1"># before</span>
+<span class="n">spark</span><span class="o">.</span><span class="n">last_day</span><span class="p">(</span><span class="n">date</span><span class="o">=</span><span class="n">d</span><span class="p">)</span> <span class="c1"># after</span>
+</pre></div>
+</div>
+</section>
<section id="changes-to-the-datafusion-python-util-crate">
<h3>Changes to the <code class="docutils literal notranslate"><span class="pre">datafusion-python-util</span></code> crate<a class="headerlink" href="#changes-to-the-datafusion-python-util-crate" title="Link to this heading">#</a></h3>
<p>Extension libraries written in Rust usually depend on the
@@ -911,6 +1060,13 @@ using the specific datafusion project crates.</p>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#extension-codecs-compose-instead-of-replacing">Extension codecs compose instead of replacing</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#capsule-getter-protocols-moved-to-datafusion-extensions">Capsule-getter protocols moved to <code class="docutils literal notranslate"><span class="pre">datafusion.extensions</span></code></a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#sessioncontext-execute-renamed-its-second-parameter"><code class="docutils literal notranslate"><span class="pre">SessionContext.execute</span></code> renamed its second parameter</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#more-aggregate-functions-accept-distinct">More aggregate functions accept <code class="docutils literal notranslate"><span class="pre">distinct</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#chaining-keeps-options-already-set">Chaining keeps options already set</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#over-keeps-options-set-on-an-aggregate"><code class="docutils literal notranslate"><span class="pre">over()</span></code> keeps options set on an aggregate</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#python-aggregate-udfs-reject-distinct">Python aggregate UDFs reject <code class="docutils literal notranslate"><span class="pre">DISTINCT</span></code></a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#percentile-functions-keep-the-sort-direction">Percentile functions keep the sort direction</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#fill-null-subset-fills-no-columns"><code class="docutils literal notranslate"><span class="pre">fill_null(subset=[])</span></code> fills no columns</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#spark-last-day-renamed-its-parameter"><code class="docutils literal notranslate"><span class="pre">spark.last_day</span></code> renamed its parameter</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#changes-to-the-datafusion-python-util-crate">Changes to the <code class="docutils literal notranslate"><span class="pre">datafusion-python-util</span></code> crate</a></li>
</ul>
</li>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org