You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/01/05 16:07:14 UTC
(datafusion-python) branch asf-site updated: Publish built docs triggered by fc5d46e9e307dba3778b5cfd7ecd465a5d63a797
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 79345e73 Publish built docs triggered by fc5d46e9e307dba3778b5cfd7ecd465a5d63a797
79345e73 is described below
commit 79345e73fabb42f2dbd78c3c6dbda59ae68dd7de
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Mon Jan 5 16:07:08 2026 +0000
Publish built docs triggered by fc5d46e9e307dba3778b5cfd7ecd465a5d63a797
---
_sources/autoapi/datafusion/context/index.rst.txt | 14 +-
.../autoapi/datafusion/dataframe/index.rst.txt | 130 +++++++++++++----
.../datafusion/dataframe_formatter/index.rst.txt | 10 +-
_sources/autoapi/datafusion/expr/index.rst.txt | 22 +--
.../autoapi/datafusion/functions/index.rst.txt | 102 +++++++-------
_sources/autoapi/datafusion/index.rst.txt | 86 +++++++-----
.../autoapi/datafusion/record_batch/index.rst.txt | 25 +++-
.../autoapi/datafusion/user_defined/index.rst.txt | 52 ++++---
.../user-guide/common-operations/joins.rst.txt | 34 ++++-
_sources/user-guide/configuration.rst.txt | 2 +
_sources/user-guide/dataframe/index.rst.txt | 116 ++++++++++++++-
_sources/user-guide/io/arrow.rst.txt | 10 +-
_sources/user-guide/sql.rst.txt | 91 +++++++++++-
autoapi/datafusion/context/index.html | 13 +-
autoapi/datafusion/dataframe/index.html | 155 +++++++++++++++++----
autoapi/datafusion/dataframe_formatter/index.html | 8 +-
autoapi/datafusion/expr/index.html | 4 +-
autoapi/datafusion/index.html | 84 +++++++----
autoapi/datafusion/record_batch/index.html | 36 ++++-
autoapi/datafusion/user_defined/index.html | 72 +++++++---
genindex.html | 20 ++-
objects.inv | Bin 8766 -> 8815 bytes
searchindex.js | 2 +-
user-guide/common-operations/aggregations.html | 32 ++---
user-guide/common-operations/basic-info.html | 46 +++---
user-guide/common-operations/functions.html | 28 ++--
user-guide/common-operations/index.html | 1 +
user-guide/common-operations/joins.html | 51 +++++++
user-guide/configuration.html | 2 +-
user-guide/dataframe/index.html | 113 ++++++++++++++-
user-guide/introduction.html | 2 +-
user-guide/io/arrow.html | 18 +--
user-guide/sql.html | 134 ++++++++++++++++--
33 files changed, 1191 insertions(+), 324 deletions(-)
diff --git a/_sources/autoapi/datafusion/context/index.rst.txt b/_sources/autoapi/datafusion/context/index.rst.txt
index 3641fe8e..50c3a9d4 100644
--- a/_sources/autoapi/datafusion/context/index.rst.txt
+++ b/_sources/autoapi/datafusion/context/index.rst.txt
@@ -866,10 +866,14 @@ Module Contents
- .. py:method:: sql(query: str, options: SQLOptions | None = None) -> datafusion.dataframe.DataFrame
+ .. py:method:: sql(query: str, options: SQLOptions | None = None, param_values: dict[str, Any] | None = None, **named_params: Any) -> datafusion.dataframe.DataFrame
Create a :py:class:`~datafusion.DataFrame` from SQL query text.
+ See the online documentation for a description of how to perform
+ parameterized substitution via either the ``param_values`` option
+ or passing in ``named_params``.
+
Note: This API implements DDL statements such as ``CREATE TABLE`` and
``CREATE VIEW`` and DML statements such as ``INSERT INTO`` with in-memory
default implementation.See
@@ -877,12 +881,15 @@ Module Contents
:param query: SQL query text.
:param options: If provided, the query will be validated against these options.
+ :param param_values: Provides substitution of scalar values in the query
+ after parsing.
+ :param named_params: Provides string or DataFrame substitution in the query string.
:returns: DataFrame representation of the SQL query.
- .. py:method:: sql_with_options(query: str, options: SQLOptions) -> datafusion.dataframe.DataFrame
+ .. py:method:: sql_with_options(query: str, options: SQLOptions, param_values: dict[str, Any] | None = None, **named_params: Any) -> datafusion.dataframe.DataFrame
Create a :py:class:`~datafusion.dataframe.DataFrame` from SQL query text.
@@ -891,6 +898,9 @@ Module Contents
:param query: SQL query text.
:param options: SQL options.
+ :param param_values: Provides substitution of scalar values in the query
+ after parsing.
+ :param named_params: Provides string or DataFrame substitution in the query string.
:returns: DataFrame representation of the SQL query.
diff --git a/_sources/autoapi/datafusion/dataframe/index.rst.txt b/_sources/autoapi/datafusion/dataframe/index.rst.txt
index 8898ceaf..f4134a83 100644
--- a/_sources/autoapi/datafusion/dataframe/index.rst.txt
+++ b/_sources/autoapi/datafusion/dataframe/index.rst.txt
@@ -49,7 +49,7 @@ Module Contents
- .. py:method:: get_default_level() -> Optional[int]
+ .. py:method:: get_default_level() -> int | None
Get the default compression level for the compression type.
@@ -96,6 +96,9 @@ Module Contents
Two dimensional table representation of data.
+ DataFrame objects are iterable; iterating over a DataFrame yields
+ :class:`datafusion.RecordBatch` instances lazily.
+
See :ref:`user_guide_concepts` in the online documentation for more information.
This constructor is not to be used by the end user.
@@ -104,24 +107,52 @@ Module Contents
create a :py:class:`DataFrame`.
+ .. py:method:: __aiter__() -> collections.abc.AsyncIterator[datafusion.record_batch.RecordBatch]
+
+ Return an async iterator over this DataFrame's record batches.
+
+ We're using __aiter__ because we support Python < 3.10 where aiter() is not
+ available.
+
+
+
.. py:method:: __arrow_c_stream__(requested_schema: object | None = None) -> object
- Export an Arrow PyCapsule Stream.
+ Export the DataFrame as an Arrow C Stream.
+
+ The DataFrame is executed using DataFusion's streaming APIs and exposed via
+ Arrow's C Stream interface. Record batches are produced incrementally, so the
+ full result set is never materialized in memory.
+
+ When ``requested_schema`` is provided, DataFusion applies only simple
+ projections such as selecting a subset of existing columns or reordering
+ them. Column renaming, computed expressions, or type coercion are not
+ supported through this interface.
- This will execute and collect the DataFrame. We will attempt to respect the
- requested schema, but only trivial transformations will be applied such as only
- returning the fields listed in the requested schema if their data types match
- those in the DataFrame.
+ :param requested_schema: Either a :py:class:`pyarrow.Schema` or an Arrow C
+ Schema capsule (``PyCapsule``) produced by
+ ``schema._export_to_c_capsule()``. The DataFrame will attempt to
+ align its output with the fields and order specified by this schema.
- :param requested_schema: Attempt to provide the DataFrame using this schema.
+ :returns: Arrow ``PyCapsule`` object representing an ``ArrowArrayStream``.
- :returns: Arrow PyCapsule object.
+ For practical usage patterns, see the Apache Arrow streaming
+ documentation: https://arrow.apache.org/docs/python/ipc.html#streaming.
+
+ For details on DataFusion's Arrow integration and DataFrame streaming,
+ see the user guide (user-guide/io/arrow and user-guide/dataframe/index).
+
+ .. rubric:: Notes
+
+ The Arrow C Data Interface PyCapsule details are documented by Apache
+ Arrow and can be found at:
+ https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html
.. py:method:: __getitem__(key: str | list[str]) -> DataFrame
- Return a new :py:class`DataFrame` with the specified column or columns.
+ Return a new :py:class:`DataFrame` with the specified column or columns.
:param key: Column name or list of column names to select.
@@ -129,6 +160,12 @@ Module Contents
+ .. py:method:: __iter__() -> collections.abc.Iterator[datafusion.record_batch.RecordBatch]
+
+ Return an iterator over this DataFrame's record batches.
+
+
+
.. py:method:: __repr__() -> str
Return a string representation of the DataFrame.
@@ -181,6 +218,12 @@ Module Contents
+ .. py:method:: collect_column(column_name: str) -> pyarrow.Array | pyarrow.ChunkedArray
+
+ Executes this :py:class:`DataFrame` for a single column.
+
+
+
.. py:method:: collect_partitioned() -> list[list[pyarrow.RecordBatch]]
Execute this DataFrame and collect all partitioned results.
@@ -328,24 +371,25 @@ Module Contents
- .. py:method:: filter(*predicates: datafusion.expr.Expr) -> DataFrame
+ .. py:method:: filter(*predicates: datafusion.expr.Expr | str) -> DataFrame
Return a DataFrame for which ``predicate`` evaluates to ``True``.
Rows for which ``predicate`` evaluates to ``False`` or ``None`` are filtered
out. If more than one predicate is provided, these predicates will be
- combined as a logical AND. Each ``predicate`` must be an
+ combined as a logical AND. Each ``predicate`` can be an
:class:`~datafusion.expr.Expr` created using helper functions such as
- :func:`datafusion.col` or :func:`datafusion.lit`.
- If more complex logic is required, see the logical operations in
- :py:mod:`~datafusion.functions`.
+ :func:`datafusion.col` or :func:`datafusion.lit`, or a SQL expression string
+ that will be parsed against the DataFrame schema. If more complex logic is
+ required, see the logical operations in :py:mod:`~datafusion.functions`.
Example::
from datafusion import col, lit
df.filter(col("a") > lit(1))
+ df.filter("a > 1")
- :param predicates: Predicate expression(s) to filter the DataFrame.
+ :param predicates: Predicate expression(s) or SQL strings to filter the DataFrame.
:returns: DataFrame after filtering.
@@ -373,7 +417,7 @@ Module Contents
- .. py:method:: into_view() -> datafusion.catalog.Table
+ .. py:method:: into_view(temporary: bool = False) -> datafusion.catalog.Table
Convert ``DataFrame`` into a :class:`~datafusion.Table`.
@@ -389,9 +433,9 @@ Module Contents
- .. py:method:: join(right: DataFrame, on: str | collections.abc.Sequence[str], how: Literal['inner', 'left', 'right', 'full', 'semi', 'anti'] = 'inner', *, left_on: None = None, right_on: None = None, join_keys: None = None) -> DataFrame
- join(right: DataFrame, on: None = None, how: Literal['inner', 'left', 'right', 'full', 'semi', 'anti'] = 'inner', *, left_on: str | collections.abc.Sequence[str], right_on: str | collections.abc.Sequence[str], join_keys: tuple[list[str], list[str]] | None = None) -> DataFrame
- join(right: DataFrame, on: None = None, how: Literal['inner', 'left', 'right', 'full', 'semi', 'anti'] = 'inner', *, join_keys: tuple[list[str], list[str]], left_on: None = None, right_on: None = None) -> DataFrame
+ .. py:method:: join(right: DataFrame, on: str | collections.abc.Sequence[str], how: Literal['inner', 'left', 'right', 'full', 'semi', 'anti'] = 'inner', *, left_on: None = None, right_on: None = None, join_keys: None = None, coalesce_duplicate_keys: bool = True) -> DataFrame
+ join(right: DataFrame, on: None = None, how: Literal['inner', 'left', 'right', 'full', 'semi', 'anti'] = 'inner', *, left_on: str | collections.abc.Sequence[str], right_on: str | collections.abc.Sequence[str], join_keys: tuple[list[str], list[str]] | None = None, coalesce_duplicate_keys: bool = True) -> DataFrame
+ join(right: DataFrame, on: None = None, how: Literal['inner', 'left', 'right', 'full', 'semi', 'anti'] = 'inner', *, join_keys: tuple[list[str], list[str]], left_on: None = None, right_on: None = None, coalesce_duplicate_keys: bool = True) -> DataFrame
Join this :py:class:`DataFrame` with another :py:class:`DataFrame`.
@@ -403,6 +447,9 @@ Module Contents
"right", "full", "semi", "anti".
:param left_on: Join column of the left dataframe.
:param right_on: Join column of the right dataframe.
+ :param coalesce_duplicate_keys: When True, coalesce the columns
+ from the right DataFrame and left DataFrame
+ that have identical names in the ``on`` fields.
:param join_keys: Tuple of two lists of column names to join on. [Deprecated]
:returns: DataFrame after join.
@@ -491,11 +538,12 @@ Module Contents
- .. py:method:: repartition_by_hash(*exprs: datafusion.expr.Expr, num: int) -> DataFrame
+ .. py:method:: repartition_by_hash(*exprs: datafusion.expr.Expr | str, num: int) -> DataFrame
Repartition a DataFrame using a hash partitioning scheme.
- :param exprs: Expressions to evaluate and perform hashing on.
+ :param exprs: Expressions or a SQL expression string to evaluate
+ and perform hashing on.
:param num: Number of partitions to repartition the DataFrame into.
:returns: Repartitioned DataFrame.
@@ -541,6 +589,17 @@ Module Contents
+ .. py:method:: select_exprs(*args: str) -> DataFrame
+
+ Project arbitrary list of expression strings into a new DataFrame.
+
+ This method will parse string expressions into logical plan expressions.
+ The output DataFrame has one column for each expression.
+
+ :returns: DataFrame only containing the specified columns.
+
+
+
.. py:method:: show(num: int = 20) -> None
Execute the DataFrame and print the result to the console.
@@ -615,7 +674,7 @@ Module Contents
- .. py:method:: transform(func: Callable[Ellipsis, DataFrame], *args: Any) -> DataFrame
+ .. py:method:: transform(func: collections.abc.Callable[Ellipsis, DataFrame], *args: Any) -> DataFrame
Apply a function to the current DataFrame which returns another DataFrame.
@@ -675,12 +734,13 @@ Module Contents
- .. py:method:: with_column(name: str, expr: datafusion.expr.Expr) -> DataFrame
+ .. py:method:: with_column(name: str, expr: datafusion.expr.Expr | str) -> DataFrame
Add an additional column to the DataFrame.
The ``expr`` must be an :class:`~datafusion.expr.Expr` constructed with
- :func:`datafusion.col` or :func:`datafusion.lit`.
+ :func:`datafusion.col` or :func:`datafusion.lit`, or a SQL expression
+ string that will be parsed against the DataFrame schema.
Example::
@@ -710,13 +770,14 @@ Module Contents
- .. py:method:: with_columns(*exprs: datafusion.expr.Expr | Iterable[datafusion.expr.Expr], **named_exprs: datafusion.expr.Expr) -> DataFrame
+ .. py:method:: with_columns(*exprs: datafusion.expr.Expr | str | collections.abc.Iterable[datafusion.expr.Expr | str], **named_exprs: datafusion.expr.Expr | str) -> DataFrame
Add columns to the DataFrame.
- By passing expressions, iterables of expressions, or named expressions.
+ By passing expressions, iterables of expressions, string SQL expressions,
+ or named expressions.
All expressions must be :class:`~datafusion.expr.Expr` objects created via
- :func:`datafusion.col` or :func:`datafusion.lit`.
+ :func:`datafusion.col` or :func:`datafusion.lit`, or SQL expression strings.
To pass named expressions use the form ``name=Expr``.
Example usage: The following will add 4 columns labeled ``a``, ``b``, ``c``,
@@ -729,7 +790,16 @@ Module Contents
d=lit(3)
)
- :param exprs: Either a single expression or an iterable of expressions to add.
+ Equivalent example using just SQL strings:
+
+ df = df.with_columns(
+ "x as a",
+ ["1 as b", "y as c"],
+ d="3"
+ )
+
+ :param exprs: Either a single expression, an iterable of expressions to add or
+ SQL expression strings.
:param named_exprs: Named expressions in the form of ``name=expr``
:returns: DataFrame with the new columns added.
@@ -849,7 +919,7 @@ Module Contents
Replacement is typically based on a unique key or primary key.
-.. py:class:: ParquetColumnOptions(encoding: Optional[str] = None, dictionary_enabled: Optional[bool] = None, compression: Optional[str] = None, statistics_enabled: Optional[str] = None, bloom_filter_enabled: Optional[bool] = None, bloom_filter_fpp: Optional[float] = None, bloom_filter_ndv: Optional[int] = None)
+.. py:class:: ParquetColumnOptions(encoding: str | None = None, dictionary_enabled: bool | None = None, compression: str | None = None, statistics_enabled: str | None = None, bloom_filter_enabled: bool | None = None, bloom_filter_fpp: float | None = None, bloom_filter_ndv: int | None = None)
Parquet options for individual columns.
@@ -916,7 +986,7 @@ Module Contents
-.. py:class:: ParquetWriterOptions(data_pagesize_limit: int = 1024 * 1024, write_batch_size: int = 1024, writer_version: str = '1.0', skip_arrow_metadata: bool = False, compression: Optional[str] = 'zstd(3)', compression_level: Optional[int] = None, dictionary_enabled: Optional[bool] = True, dictionary_page_size_limit: int = 1024 * 1024, statistics_enabled: Optional[str] = 'page', max_row_group_size: int = 1024 * 1024, created_by: str = 'datafusion-python', column_index_truncate_length: [...]
+.. py:class:: ParquetWriterOptions(data_pagesize_limit: int = 1024 * 1024, write_batch_size: int = 1024, writer_version: str = '1.0', skip_arrow_metadata: bool = False, compression: str | None = 'zstd(3)', compression_level: int | None = None, dictionary_enabled: bool | None = True, dictionary_page_size_limit: int = 1024 * 1024, statistics_enabled: str | None = 'page', max_row_group_size: int = 1024 * 1024, created_by: str = 'datafusion-python', column_index_truncate_length: int | None = [...]
Advanced parquet writer options.
diff --git a/_sources/autoapi/datafusion/dataframe_formatter/index.rst.txt b/_sources/autoapi/datafusion/dataframe_formatter/index.rst.txt
index ad3e62b5..7f5b3f53 100644
--- a/_sources/autoapi/datafusion/dataframe_formatter/index.rst.txt
+++ b/_sources/autoapi/datafusion/dataframe_formatter/index.rst.txt
@@ -52,7 +52,7 @@ Module Contents
-.. py:class:: DataFrameHtmlFormatter(max_cell_length: int = 25, max_width: int = 1000, max_height: int = 300, max_memory_bytes: int = 2 * 1024 * 1024, min_rows_display: int = 20, repr_rows: int = 10, enable_cell_expansion: bool = True, custom_css: Optional[str] = None, show_truncation_message: bool = True, style_provider: Optional[StyleProvider] = None, use_shared_styles: bool = True)
+.. py:class:: DataFrameHtmlFormatter(max_cell_length: int = 25, max_width: int = 1000, max_height: int = 300, max_memory_bytes: int = 2 * 1024 * 1024, min_rows_display: int = 20, repr_rows: int = 10, enable_cell_expansion: bool = True, custom_css: str | None = None, show_truncation_message: bool = True, style_provider: StyleProvider | None = None, use_shared_styles: bool = True)
Configurable HTML formatter for DataFusion DataFrames.
@@ -237,7 +237,7 @@ Module Contents
- .. py:method:: set_custom_cell_builder(builder: Callable[[Any, int, int, str], str]) -> None
+ .. py:method:: set_custom_cell_builder(builder: collections.abc.Callable[[Any, int, int, str], str]) -> None
Set a custom cell builder function.
@@ -245,7 +245,7 @@ Module Contents
- .. py:method:: set_custom_header_builder(builder: Callable[[Any], str]) -> None
+ .. py:method:: set_custom_header_builder(builder: collections.abc.Callable[[Any], str]) -> None
Set a custom header builder function.
@@ -254,13 +254,13 @@ Module Contents
.. py:attribute:: _custom_cell_builder
- :type: Optional[Callable[[Any, int, int, str], str]]
+ :type: collections.abc.Callable[[Any, int, int, str], str] | None
:value: None
.. py:attribute:: _custom_header_builder
- :type: Optional[Callable[[Any], str]]
+ :type: collections.abc.Callable[[Any], str] | None
:value: None
diff --git a/_sources/autoapi/datafusion/expr/index.rst.txt b/_sources/autoapi/datafusion/expr/index.rst.txt
index 71f68f7e..2bea04f4 100644
--- a/_sources/autoapi/datafusion/expr/index.rst.txt
+++ b/_sources/autoapi/datafusion/expr/index.rst.txt
@@ -339,7 +339,7 @@ Module Contents
- .. py:method:: alias(name: str, metadata: Optional[dict[str, str]] = None) -> Expr
+ .. py:method:: alias(name: str, metadata: dict[str, str] | None = None) -> Expr
Assign a name to the expression.
@@ -462,7 +462,7 @@ Module Contents
- .. py:method:: cast(to: Union[pyarrow.DataType[Any], type]) -> Expr
+ .. py:method:: cast(to: pyarrow.DataType[Any] | type) -> Expr
Cast to a new data type.
@@ -572,13 +572,13 @@ Module Contents
- .. py:method:: fill_nan(value: Optional[Union[Any, Expr]] = None) -> Expr
+ .. py:method:: fill_nan(value: Any | Expr | None = None) -> Expr
Fill NaN values with a provided value.
- .. py:method:: fill_null(value: Optional[Union[Any, Expr]] = None) -> Expr
+ .. py:method:: fill_null(value: Any | Expr | None = None) -> Expr
Fill NULL values with a provided value.
@@ -756,7 +756,7 @@ Module Contents
- .. py:method:: order_by(*exprs: Union[Expr, SortExpr]) -> ExprFuncBuilder
+ .. py:method:: order_by(*exprs: Expr | SortExpr) -> ExprFuncBuilder
Set the ordering for a window or aggregate function.
@@ -790,7 +790,7 @@ Module Contents
.. py:method:: python_value() -> Any
- Extracts the Expr value into a PyObject.
+ Extracts the Expr value into `Any`.
This is only valid for literal expressions.
@@ -1048,7 +1048,7 @@ Module Contents
.. py:attribute:: raw_sort
-.. py:class:: Window(partition_by: Optional[Union[list[Expr], Expr]] = None, window_frame: Optional[WindowFrame] = None, order_by: Optional[Union[list[Union[SortExpr, Expr, str]], Expr, SortExpr, str]] = None, null_treatment: Optional[datafusion.common.NullTreatment] = None)
+.. py:class:: Window(partition_by: list[Expr] | Expr | None = None, window_frame: WindowFrame | None = None, order_by: list[SortExpr | Expr | str] | Expr | SortExpr | str | None = None, null_treatment: datafusion.common.NullTreatment | None = None)
Define reusable window parameters.
@@ -1080,7 +1080,7 @@ Module Contents
-.. py:class:: WindowFrame(units: str, start_bound: Optional[Any], end_bound: Optional[Any])
+.. py:class:: WindowFrame(units: str, start_bound: Any | None, end_bound: Any | None)
Defines a window frame for performing window operations.
@@ -1131,7 +1131,7 @@ Module Contents
Constructs a window frame bound.
- .. py:method:: get_offset() -> Optional[int]
+ .. py:method:: get_offset() -> int | None
Returns the offset of the window frame.
@@ -1164,7 +1164,7 @@ Module Contents
.. py:attribute:: frame_bound
-.. py:function:: ensure_expr(value: Union[Expr, Any]) -> datafusion._internal.expr.Expr
+.. py:function:: ensure_expr(value: Expr | Any) -> datafusion._internal.expr.Expr
Return the internal expression from ``Expr`` or raise ``TypeError``.
@@ -1179,7 +1179,7 @@ Module Contents
:raises TypeError: If ``value`` is not an instance of :class:`Expr`.
-.. py:function:: ensure_expr_list(exprs: Iterable[Union[Expr, Iterable[Expr]]]) -> list[datafusion._internal.expr.Expr]
+.. py:function:: ensure_expr_list(exprs: collections.abc.Iterable[Expr | collections.abc.Iterable[Expr]]) -> list[datafusion._internal.expr.Expr]
Flatten an iterable of expressions, validating each via ``ensure_expr``.
diff --git a/_sources/autoapi/datafusion/functions/index.rst.txt b/_sources/autoapi/datafusion/functions/index.rst.txt
index 6a2a049a..10b7f1a0 100644
--- a/_sources/autoapi/datafusion/functions/index.rst.txt
+++ b/_sources/autoapi/datafusion/functions/index.rst.txt
@@ -285,7 +285,7 @@ Module Contents
Returns inverse hyperbolic cosine.
-.. py:function:: alias(expr: datafusion.expr.Expr, name: str, metadata: Optional[dict[str, str]] = None) -> datafusion.expr.Expr
+.. py:function:: alias(expr: datafusion.expr.Expr, name: str, metadata: dict[str, str] | None = None) -> datafusion.expr.Expr
Creates an alias expression with an optional metadata dictionary.
@@ -296,7 +296,7 @@ Module Contents
:returns: An expression with the given alias
-.. py:function:: approx_distinct(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: approx_distinct(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the approximate number of distinct values.
@@ -311,7 +311,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: approx_median(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: approx_median(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the approximate median value.
@@ -325,7 +325,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: approx_percentile_cont(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, percentile: float, num_centroids: Optional[int] = None, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: approx_percentile_cont(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, percentile: float, num_centroids: int | None = None, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the value that is approximately at a given percentile of ``expr``.
@@ -348,7 +348,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: approx_percentile_cont_with_weight(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, weight: datafusion.expr.Expr, percentile: float, num_centroids: Optional[int] = None, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: approx_percentile_cont_with_weight(sort_expression: datafusion.expr.Expr | datafusion.expr.SortExpr, weight: datafusion.expr.Expr, percentile: float, num_centroids: int | None = None, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the value of the weighted approximate percentile.
@@ -372,7 +372,7 @@ Module Contents
This is an alias for :py:func:`make_array`.
-.. py:function:: array_agg(expression: datafusion.expr.Expr, distinct: bool = False, filter: Optional[datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: array_agg(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Aggregate values into an array.
@@ -643,7 +643,7 @@ Module Contents
Returns inverse hyperbolic tangent.
-.. py:function:: avg(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: avg(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the average value.
@@ -656,7 +656,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: bit_and(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: bit_and(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the bitwise AND of the argument.
@@ -674,7 +674,7 @@ Module Contents
Returns the number of bits in the string argument.
-.. py:function:: bit_or(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: bit_or(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the bitwise OR of the argument.
@@ -687,7 +687,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: bit_xor(expression: datafusion.expr.Expr, distinct: bool = False, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: bit_xor(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the bitwise XOR of the argument.
@@ -701,7 +701,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: bool_and(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: bool_and(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the boolean AND of the argument.
@@ -715,7 +715,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: bool_or(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: bool_or(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the boolean OR of the argument.
@@ -797,7 +797,7 @@ Module Contents
``NULL`` arguments are ignored. ``separator`` should not be ``NULL``.
-.. py:function:: corr(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: corr(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the correlation coefficient between ``value1`` and ``value2``.
@@ -826,7 +826,7 @@ Module Contents
Returns the cotangent of the argument.
-.. py:function:: count(expressions: datafusion.expr.Expr | list[datafusion.expr.Expr] | None = None, distinct: bool = False, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: count(expressions: datafusion.expr.Expr | list[datafusion.expr.Expr] | None = None, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the number of rows that match the given arguments.
@@ -840,7 +840,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: count_star(filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: count_star(filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Create a COUNT(1) aggregate expression.
@@ -852,14 +852,14 @@ Module Contents
:param filter: If provided, only count rows for which the filter is True
-.. py:function:: covar(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: covar(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sample covariance.
This is an alias for :py:func:`covar_samp`.
-.. py:function:: covar_pop(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: covar_pop(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the population covariance.
@@ -873,7 +873,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: covar_samp(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: covar_samp(value_y: datafusion.expr.Expr, value_x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sample covariance.
@@ -887,7 +887,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: cume_dist(partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: cume_dist(partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a cumulative distribution window function.
@@ -963,7 +963,7 @@ Module Contents
Converts the argument from radians to degrees.
-.. py:function:: dense_rank(partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: dense_rank(partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a dense_rank window function.
@@ -1039,7 +1039,7 @@ Module Contents
The string list is a string composed of substrings separated by ``,`` characters.
-.. py:function:: first_value(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None, null_treatment: datafusion.common.NullTreatment = NullTreatment.RESPECT_NULLS) -> datafusion.expr.Expr
+.. py:function:: first_value(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment = NullTreatment.RESPECT_NULLS) -> datafusion.expr.Expr
Returns the first value in a group of values.
@@ -1102,7 +1102,7 @@ Module Contents
Returns true if a given number is +0.0 or -0.0 otherwise returns false.
-.. py:function:: lag(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Optional[Any] = None, partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: lag(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a lag window function.
@@ -1135,7 +1135,7 @@ Module Contents
lag(col("b"), order_by="ts")
-.. py:function:: last_value(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None, null_treatment: datafusion.common.NullTreatment = NullTreatment.RESPECT_NULLS) -> datafusion.expr.Expr
+.. py:function:: last_value(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment = NullTreatment.RESPECT_NULLS) -> datafusion.expr.Expr
Returns the last value in a group of values.
@@ -1160,7 +1160,7 @@ Module Contents
Returns the least common multiple.
-.. py:function:: lead(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Optional[Any] = None, partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: lead(arg: datafusion.expr.Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a lead window function.
@@ -1481,7 +1481,7 @@ Module Contents
This is an alias for :py:func:`make_array`.
-.. py:function:: max(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: max(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Aggregate function that returns the maximum value of the argument.
@@ -1497,14 +1497,14 @@ Module Contents
Computes an MD5 128-bit checksum for a string expression.
-.. py:function:: mean(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: mean(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the average (mean) value of the argument.
This is an alias for :py:func:`avg`.
-.. py:function:: median(expression: datafusion.expr.Expr, distinct: bool = False, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: median(expression: datafusion.expr.Expr, distinct: bool = False, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the median of a set of numbers.
@@ -1519,7 +1519,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: min(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: min(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Returns the minimum value of the argument.
@@ -1547,7 +1547,7 @@ Module Contents
This will use the same value for all instances of now() in same statement.
-.. py:function:: nth_value(expression: datafusion.expr.Expr, n: int, filter: Optional[datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None, null_treatment: datafusion.common.NullTreatment = NullTreatment.RESPECT_NULLS) -> datafusion.expr.Expr
+.. py:function:: nth_value(expression: datafusion.expr.Expr, n: int, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None, null_treatment: datafusion.common.NullTreatment = NullTreatment.RESPECT_NULLS) -> datafusion.expr.Expr
Returns the n-th value in a group of values.
@@ -1568,7 +1568,7 @@ Module Contents
df.select(nth_value(col("a"), 2, order_by="ts"))
-.. py:function:: ntile(groups: int, partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: ntile(groups: int, partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a n-tile window function.
@@ -1628,7 +1628,7 @@ Module Contents
extends for ``length`` characters with new substring.
-.. py:function:: percent_rank(partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: percent_rank(partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a percent_rank window function.
@@ -1687,7 +1687,7 @@ Module Contents
Create a list of values in the range between start and stop.
-.. py:function:: rank(partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: rank(partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a rank window function.
@@ -1753,7 +1753,7 @@ Module Contents
<https://docs.rs/regex/latest/regex/#grouping-and-flags>
-.. py:function:: regr_avgx(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_avgx(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the average of the independent variable ``x``.
@@ -1768,7 +1768,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_avgy(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_avgy(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the average of the dependent variable ``y``.
@@ -1783,7 +1783,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_count(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_count(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Counts the number of rows in which both expressions are not null.
@@ -1798,7 +1798,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_intercept(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_intercept(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the intercept from the linear regression.
@@ -1813,7 +1813,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_r2(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_r2(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the R-squared value from linear regression.
@@ -1828,7 +1828,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_slope(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_slope(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the slope from linear regression.
@@ -1843,7 +1843,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_sxx(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_sxx(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sum of squares of the independent variable ``x``.
@@ -1858,7 +1858,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_sxy(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_sxy(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sum of products of pairs of numbers.
@@ -1873,7 +1873,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: regr_syy(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: regr_syy(y: datafusion.expr.Expr, x: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sum of squares of the dependent variable ``y``.
@@ -1917,7 +1917,7 @@ Module Contents
``round(lit(125.2345), lit(-2))`` would yield a value of ``100.0``.
-.. py:function:: row_number(partition_by: Optional[list[datafusion.expr.Expr] | datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: row_number(partition_by: list[datafusion.expr.Expr] | datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Create a row number window function.
@@ -2009,7 +2009,7 @@ Module Contents
Returns true if string starts with prefix.
-.. py:function:: stddev(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: stddev(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the standard deviation of the argument.
@@ -2020,7 +2020,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: stddev_pop(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: stddev_pop(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the population standard deviation of the argument.
@@ -2031,14 +2031,14 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: stddev_samp(arg: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: stddev_samp(arg: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sample standard deviation of the argument.
This is an alias for :py:func:`stddev`.
-.. py:function:: string_agg(expression: datafusion.expr.Expr, delimiter: str, filter: Optional[datafusion.expr.Expr] = None, order_by: Optional[list[datafusion.expr.SortKey] | datafusion.expr.SortKey] = None) -> datafusion.expr.Expr
+.. py:function:: string_agg(expression: datafusion.expr.Expr, delimiter: str, filter: datafusion.expr.Expr | None = None, order_by: list[datafusion.expr.SortKey] | datafusion.expr.SortKey | None = None) -> datafusion.expr.Expr
Concatenates the input strings.
@@ -2088,7 +2088,7 @@ Module Contents
Substring from the ``position`` with ``length`` characters.
-.. py:function:: sum(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: sum(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sum of a set of numbers.
@@ -2183,14 +2183,14 @@ Module Contents
Returns uuid v4 as a string value.
-.. py:function:: var(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: var(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sample variance of the argument.
This is an alias for :py:func:`var_samp`.
-.. py:function:: var_pop(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: var_pop(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the population variance of the argument.
@@ -2201,7 +2201,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: var_samp(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: var_samp(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sample variance of the argument.
@@ -2212,7 +2212,7 @@ Module Contents
:param filter: If provided, only compute against rows for which the filter is True
-.. py:function:: var_sample(expression: datafusion.expr.Expr, filter: Optional[datafusion.expr.Expr] = None) -> datafusion.expr.Expr
+.. py:function:: var_sample(expression: datafusion.expr.Expr, filter: datafusion.expr.Expr | None = None) -> datafusion.expr.Expr
Computes the sample variance of the argument.
diff --git a/_sources/autoapi/datafusion/index.rst.txt b/_sources/autoapi/datafusion/index.rst.txt
index e4208b05..59e6d291 100644
--- a/_sources/autoapi/datafusion/index.rst.txt
+++ b/_sources/autoapi/datafusion/index.rst.txt
@@ -131,7 +131,8 @@ Package Contents
-.. py:class:: AggregateUDF(name: str, accumulator: Callable[[], Accumulator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str)
+.. py:class:: AggregateUDF(name: str, accumulator: collections.abc.Callable[[], Accumulator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str)
+ AggregateUDF(name: str, accumulator: AggregateUDFExportable, input_types: None = ..., return_type: None = ..., state_type: None = ..., volatility: None = ...)
Class for performing scalar user-defined functions (UDF).
@@ -159,7 +160,7 @@ Package Contents
- .. py:method:: from_pycapsule(func: AggregateUDFExportable) -> AggregateUDF
+ .. py:method:: from_pycapsule(func: AggregateUDFExportable | _typeshed.CapsuleType) -> AggregateUDF
:staticmethod:
@@ -170,8 +171,10 @@ Package Contents
- .. py:method:: udaf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, AggregateUDF]
- udaf(accum: Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: Optional[str] = None) -> AggregateUDF
+ .. py:method:: udaf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, AggregateUDF]
+ udaf(accum: collections.abc.Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) -> AggregateUDF
+ udaf(accum: AggregateUDFExportable) -> AggregateUDF
+ udaf(accum: _typeshed.CapsuleType) -> AggregateUDF
:staticmethod:
@@ -564,7 +567,7 @@ Package Contents
- .. py:method:: alias(name: str, metadata: Optional[dict[str, str]] = None) -> Expr
+ .. py:method:: alias(name: str, metadata: dict[str, str] | None = None) -> Expr
Assign a name to the expression.
@@ -687,7 +690,7 @@ Package Contents
- .. py:method:: cast(to: Union[pyarrow.DataType[Any], type]) -> Expr
+ .. py:method:: cast(to: pyarrow.DataType[Any] | type) -> Expr
Cast to a new data type.
@@ -797,13 +800,13 @@ Package Contents
- .. py:method:: fill_nan(value: Optional[Union[Any, Expr]] = None) -> Expr
+ .. py:method:: fill_nan(value: Any | Expr | None = None) -> Expr
Fill NaN values with a provided value.
- .. py:method:: fill_null(value: Optional[Union[Any, Expr]] = None) -> Expr
+ .. py:method:: fill_null(value: Any | Expr | None = None) -> Expr
Fill NULL values with a provided value.
@@ -981,7 +984,7 @@ Package Contents
- .. py:method:: order_by(*exprs: Union[Expr, SortExpr]) -> ExprFuncBuilder
+ .. py:method:: order_by(*exprs: Expr | SortExpr) -> ExprFuncBuilder
Set the ordering for a window or aggregate function.
@@ -1015,7 +1018,7 @@ Package Contents
.. py:method:: python_value() -> Any
- Extracts the Expr value into a PyObject.
+ Extracts the Expr value into `Any`.
This is only valid for literal expressions.
@@ -1351,7 +1354,7 @@ Package Contents
.. py:attribute:: _raw_plan
-.. py:class:: ParquetColumnOptions(encoding: Optional[str] = None, dictionary_enabled: Optional[bool] = None, compression: Optional[str] = None, statistics_enabled: Optional[str] = None, bloom_filter_enabled: Optional[bool] = None, bloom_filter_fpp: Optional[float] = None, bloom_filter_ndv: Optional[int] = None)
+.. py:class:: ParquetColumnOptions(encoding: str | None = None, dictionary_enabled: bool | None = None, compression: str | None = None, statistics_enabled: str | None = None, bloom_filter_enabled: bool | None = None, bloom_filter_fpp: float | None = None, bloom_filter_ndv: int | None = None)
Parquet options for individual columns.
@@ -1418,7 +1421,7 @@ Package Contents
-.. py:class:: ParquetWriterOptions(data_pagesize_limit: int = 1024 * 1024, write_batch_size: int = 1024, writer_version: str = '1.0', skip_arrow_metadata: bool = False, compression: Optional[str] = 'zstd(3)', compression_level: Optional[int] = None, dictionary_enabled: Optional[bool] = True, dictionary_page_size_limit: int = 1024 * 1024, statistics_enabled: Optional[str] = 'page', max_row_group_size: int = 1024 * 1024, created_by: str = 'datafusion-python', column_index_truncate_length: [...]
+.. py:class:: ParquetWriterOptions(data_pagesize_limit: int = 1024 * 1024, write_batch_size: int = 1024, writer_version: str = '1.0', skip_arrow_metadata: bool = False, compression: str | None = 'zstd(3)', compression_level: int | None = None, dictionary_enabled: bool | None = True, dictionary_page_size_limit: int = 1024 * 1024, statistics_enabled: str | None = 'page', max_row_group_size: int = 1024 * 1024, created_by: str = 'datafusion-python', column_index_truncate_length: int | None = [...]
Advanced parquet writer options.
@@ -1601,6 +1604,23 @@ Package Contents
See the :py:class:`RecordBatchStream` iterator for generating this class.
+ .. py:method:: __arrow_c_array__(requested_schema: object | None = None) -> tuple[object, object]
+
+ Export the record batch via the Arrow C Data Interface.
+
+ This allows zero-copy interchange with libraries that support the
+ `Arrow PyCapsule interface <https://arrow.apache.org/docs/format/
+ CDataInterface/PyCapsuleInterface.html>`_.
+
+ :param requested_schema: Attempt to provide the record batch using this
+ schema. Only straightforward projections such as column
+ selection or reordering are applied.
+
+ :returns: Two Arrow PyCapsule objects representing the ``ArrowArray`` and
+ ``ArrowSchema``.
+
+
+
.. py:method:: to_pyarrow() -> pyarrow.RecordBatch
Convert to :py:class:`pa.RecordBatch`.
@@ -1622,7 +1642,7 @@ Package Contents
.. py:method:: __aiter__() -> typing_extensions.Self
- Async iterator function.
+ Return an asynchronous iterator over record batches.
@@ -1630,19 +1650,19 @@ Package Contents
:async:
- Async iterator function.
+ Return the next :py:class:`RecordBatch` in the stream asynchronously.
.. py:method:: __iter__() -> typing_extensions.Self
- Iterator function.
+ Return an iterator over record batches.
.. py:method:: __next__() -> RecordBatch
- Iterator function.
+ Return the next :py:class:`RecordBatch` in the stream.
@@ -1821,7 +1841,7 @@ Package Contents
.. py:attribute:: options_internal
-.. py:class:: ScalarUDF(name: str, func: Callable[Ellipsis, _R], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: _R, volatility: Volatility | str)
+.. py:class:: ScalarUDF(name: str, func: collections.abc.Callable[Ellipsis, _R], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: _R, volatility: Volatility | str)
Class for performing scalar user-defined functions (UDF).
@@ -1859,8 +1879,8 @@ Package Contents
- .. py:method:: udf(input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, ScalarUDF]
- udf(func: Callable[Ellipsis, _R], input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: Optional[str] = None) -> ScalarUDF
+ .. py:method:: udf(input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, ScalarUDF]
+ udf(func: collections.abc.Callable[Ellipsis, _R], input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: str | None = None) -> ScalarUDF
udf(func: ScalarUDFExportable) -> ScalarUDF
:staticmethod:
@@ -2120,7 +2140,7 @@ Package Contents
Returns the schema associated with this table.
-.. py:class:: TableFunction(name: str, func: Callable[[], any])
+.. py:class:: TableFunction(name: str, func: collections.abc.Callable[[], any])
Class for performing user-defined table functions (UDTF).
@@ -2145,7 +2165,7 @@ Package Contents
- .. py:method:: _create_table_udf(func: Callable[Ellipsis, Any], name: str) -> TableFunction
+ .. py:method:: _create_table_udf(func: collections.abc.Callable[Ellipsis, Any], name: str) -> TableFunction
:staticmethod:
@@ -2153,7 +2173,7 @@ Package Contents
- .. py:method:: _create_table_udf_decorator(name: Optional[str] = None) -> Callable[[Callable[[], WindowEvaluator]], Callable[Ellipsis, datafusion.expr.Expr]]
+ .. py:method:: _create_table_udf_decorator(name: str | None = None) -> collections.abc.Callable[[collections.abc.Callable[[], WindowEvaluator]], collections.abc.Callable[Ellipsis, datafusion.expr.Expr]]
:staticmethod:
@@ -2161,8 +2181,8 @@ Package Contents
- .. py:method:: udtf(name: str) -> Callable[Ellipsis, Any]
- udtf(func: Callable[[], Any], name: str) -> TableFunction
+ .. py:method:: udtf(name: str) -> collections.abc.Callable[Ellipsis, Any]
+ udtf(func: collections.abc.Callable[[], Any], name: str) -> TableFunction
:staticmethod:
@@ -2173,7 +2193,7 @@ Package Contents
.. py:attribute:: _udtf
-.. py:class:: WindowFrame(units: str, start_bound: Optional[Any], end_bound: Optional[Any])
+.. py:class:: WindowFrame(units: str, start_bound: Any | None, end_bound: Any | None)
Defines a window frame for performing window operations.
@@ -2215,7 +2235,7 @@ Package Contents
.. py:attribute:: window_frame
-.. py:class:: WindowUDF(name: str, func: Callable[[], WindowEvaluator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str)
+.. py:class:: WindowUDF(name: str, func: collections.abc.Callable[[], WindowEvaluator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str)
Class for performing window user-defined functions (UDF).
@@ -2243,7 +2263,7 @@ Package Contents
- .. py:method:: _create_window_udf(func: Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> WindowUDF
+ .. py:method:: _create_window_udf(func: collections.abc.Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> WindowUDF
:staticmethod:
@@ -2251,7 +2271,7 @@ Package Contents
- .. py:method:: _create_window_udf_decorator(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> Callable[[Callable[[], WindowEvaluator]], Callable[Ellipsis, datafusion.expr.Expr]]
+ .. py:method:: _create_window_udf_decorator(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[[collections.abc.Callable[[], WindowEvaluator]], collections.abc.Callable[Ellipsis, datafusion.expr.Expr]]
:staticmethod:
@@ -2259,7 +2279,7 @@ Package Contents
- .. py:method:: _get_default_name(func: Callable) -> str
+ .. py:method:: _get_default_name(func: collections.abc.Callable) -> str
:staticmethod:
@@ -2286,8 +2306,8 @@ Package Contents
- .. py:method:: udwf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, WindowUDF]
- udwf(func: Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> WindowUDF
+ .. py:method:: udwf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, WindowUDF]
+ udwf(func: collections.abc.Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> WindowUDF
:staticmethod:
@@ -2366,12 +2386,12 @@ Package Contents
... )
-.. py:function:: lit(value) -> expr.Expr
+.. py:function:: lit(value: Any) -> expr.Expr
Create a literal expression.
-.. py:function:: literal(value) -> expr.Expr
+.. py:function:: literal(value: Any) -> expr.Expr
Create a literal expression.
diff --git a/_sources/autoapi/datafusion/record_batch/index.rst.txt b/_sources/autoapi/datafusion/record_batch/index.rst.txt
index ae91b4c6..f51a7e66 100644
--- a/_sources/autoapi/datafusion/record_batch/index.rst.txt
+++ b/_sources/autoapi/datafusion/record_batch/index.rst.txt
@@ -33,6 +33,23 @@ Module Contents
See the :py:class:`RecordBatchStream` iterator for generating this class.
+ .. py:method:: __arrow_c_array__(requested_schema: object | None = None) -> tuple[object, object]
+
+ Export the record batch via the Arrow C Data Interface.
+
+ This allows zero-copy interchange with libraries that support the
+ `Arrow PyCapsule interface <https://arrow.apache.org/docs/format/
+ CDataInterface/PyCapsuleInterface.html>`_.
+
+ :param requested_schema: Attempt to provide the record batch using this
+ schema. Only straightforward projections such as column
+ selection or reordering are applied.
+
+ :returns: Two Arrow PyCapsule objects representing the ``ArrowArray`` and
+ ``ArrowSchema``.
+
+
+
.. py:method:: to_pyarrow() -> pyarrow.RecordBatch
Convert to :py:class:`pa.RecordBatch`.
@@ -54,7 +71,7 @@ Module Contents
.. py:method:: __aiter__() -> typing_extensions.Self
- Async iterator function.
+ Return an asynchronous iterator over record batches.
@@ -62,19 +79,19 @@ Module Contents
:async:
- Async iterator function.
+ Return the next :py:class:`RecordBatch` in the stream asynchronously.
.. py:method:: __iter__() -> typing_extensions.Self
- Iterator function.
+ Return an iterator over record batches.
.. py:method:: __next__() -> RecordBatch
- Iterator function.
+ Return the next :py:class:`RecordBatch` in the stream.
diff --git a/_sources/autoapi/datafusion/user_defined/index.rst.txt b/_sources/autoapi/datafusion/user_defined/index.rst.txt
index 591d4cda..784a39c7 100644
--- a/_sources/autoapi/datafusion/user_defined/index.rst.txt
+++ b/_sources/autoapi/datafusion/user_defined/index.rst.txt
@@ -38,6 +38,14 @@ Classes
datafusion.user_defined.WindowUDFExportable
+Functions
+---------
+
+.. autoapisummary::
+
+ datafusion.user_defined._is_pycapsule
+
+
Module Contents
---------------
@@ -78,7 +86,8 @@ Module Contents
-.. py:class:: AggregateUDF(name: str, accumulator: Callable[[], Accumulator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str)
+.. py:class:: AggregateUDF(name: str, accumulator: collections.abc.Callable[[], Accumulator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str)
+ AggregateUDF(name: str, accumulator: AggregateUDFExportable, input_types: None = ..., return_type: None = ..., state_type: None = ..., volatility: None = ...)
Class for performing scalar user-defined functions (UDF).
@@ -106,7 +115,7 @@ Module Contents
- .. py:method:: from_pycapsule(func: AggregateUDFExportable) -> AggregateUDF
+ .. py:method:: from_pycapsule(func: AggregateUDFExportable | _typeshed.CapsuleType) -> AggregateUDF
:staticmethod:
@@ -117,8 +126,10 @@ Module Contents
- .. py:method:: udaf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, AggregateUDF]
- udaf(accum: Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: Optional[str] = None) -> AggregateUDF
+ .. py:method:: udaf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, AggregateUDF]
+ udaf(accum: collections.abc.Callable[[], Accumulator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, state_type: list[pyarrow.DataType], volatility: Volatility | str, name: str | None = None) -> AggregateUDF
+ udaf(accum: AggregateUDFExportable) -> AggregateUDF
+ udaf(accum: _typeshed.CapsuleType) -> AggregateUDF
:staticmethod:
@@ -206,7 +217,7 @@ Module Contents
.. py:method:: __datafusion_aggregate_udf__() -> object
-.. py:class:: ScalarUDF(name: str, func: Callable[Ellipsis, _R], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: _R, volatility: Volatility | str)
+.. py:class:: ScalarUDF(name: str, func: collections.abc.Callable[Ellipsis, _R], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: _R, volatility: Volatility | str)
Class for performing scalar user-defined functions (UDF).
@@ -244,8 +255,8 @@ Module Contents
- .. py:method:: udf(input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, ScalarUDF]
- udf(func: Callable[Ellipsis, _R], input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: Optional[str] = None) -> ScalarUDF
+ .. py:method:: udf(input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, ScalarUDF]
+ udf(func: collections.abc.Callable[Ellipsis, _R], input_types: list[pyarrow.DataType], return_type: _R, volatility: Volatility | str, name: str | None = None) -> ScalarUDF
udf(func: ScalarUDFExportable) -> ScalarUDF
:staticmethod:
@@ -308,7 +319,7 @@ Module Contents
.. py:method:: __datafusion_scalar_udf__() -> object
-.. py:class:: TableFunction(name: str, func: Callable[[], any])
+.. py:class:: TableFunction(name: str, func: collections.abc.Callable[[], any])
Class for performing user-defined table functions (UDTF).
@@ -333,7 +344,7 @@ Module Contents
- .. py:method:: _create_table_udf(func: Callable[Ellipsis, Any], name: str) -> TableFunction
+ .. py:method:: _create_table_udf(func: collections.abc.Callable[Ellipsis, Any], name: str) -> TableFunction
:staticmethod:
@@ -341,7 +352,7 @@ Module Contents
- .. py:method:: _create_table_udf_decorator(name: Optional[str] = None) -> Callable[[Callable[[], WindowEvaluator]], Callable[Ellipsis, datafusion.expr.Expr]]
+ .. py:method:: _create_table_udf_decorator(name: str | None = None) -> collections.abc.Callable[[collections.abc.Callable[[], WindowEvaluator]], collections.abc.Callable[Ellipsis, datafusion.expr.Expr]]
:staticmethod:
@@ -349,8 +360,8 @@ Module Contents
- .. py:method:: udtf(name: str) -> Callable[Ellipsis, Any]
- udtf(func: Callable[[], Any], name: str) -> TableFunction
+ .. py:method:: udtf(name: str) -> collections.abc.Callable[Ellipsis, Any]
+ udtf(func: collections.abc.Callable[[], Any], name: str) -> TableFunction
:staticmethod:
@@ -589,7 +600,7 @@ Module Contents
-.. py:class:: WindowUDF(name: str, func: Callable[[], WindowEvaluator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str)
+.. py:class:: WindowUDF(name: str, func: collections.abc.Callable[[], WindowEvaluator], input_types: list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str)
Class for performing window user-defined functions (UDF).
@@ -617,7 +628,7 @@ Module Contents
- .. py:method:: _create_window_udf(func: Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> WindowUDF
+ .. py:method:: _create_window_udf(func: collections.abc.Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> WindowUDF
:staticmethod:
@@ -625,7 +636,7 @@ Module Contents
- .. py:method:: _create_window_udf_decorator(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> Callable[[Callable[[], WindowEvaluator]], Callable[Ellipsis, datafusion.expr.Expr]]
+ .. py:method:: _create_window_udf_decorator(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[[collections.abc.Callable[[], WindowEvaluator]], collections.abc.Callable[Ellipsis, datafusion.expr.Expr]]
:staticmethod:
@@ -633,7 +644,7 @@ Module Contents
- .. py:method:: _get_default_name(func: Callable) -> str
+ .. py:method:: _get_default_name(func: collections.abc.Callable) -> str
:staticmethod:
@@ -660,8 +671,8 @@ Module Contents
- .. py:method:: udwf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> Callable[Ellipsis, WindowUDF]
- udwf(func: Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: Optional[str] = None) -> WindowUDF
+ .. py:method:: udwf(input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> collections.abc.Callable[Ellipsis, WindowUDF]
+ udwf(func: collections.abc.Callable[[], WindowEvaluator], input_types: pyarrow.DataType | list[pyarrow.DataType], return_type: pyarrow.DataType, volatility: Volatility | str, name: str | None = None) -> WindowUDF
:staticmethod:
@@ -728,6 +739,11 @@ Module Contents
.. py:method:: __datafusion_window_udf__() -> object
+.. py:function:: _is_pycapsule(value: object) -> TypeGuard[_typeshed.CapsuleType]
+
+ Return ``True`` when ``value`` is a CPython ``PyCapsule``.
+
+
.. py:data:: _R
.. py:data:: udaf
diff --git a/_sources/user-guide/common-operations/joins.rst.txt b/_sources/user-guide/common-operations/joins.rst.txt
index 40d92215..1d9d7038 100644
--- a/_sources/user-guide/common-operations/joins.rst.txt
+++ b/_sources/user-guide/common-operations/joins.rst.txt
@@ -101,4 +101,36 @@ the right table.
.. ipython:: python
- left.join(right, left_on="customer_id", right_on="id", how="anti")
\ No newline at end of file
+ left.join(right, left_on="customer_id", right_on="id", how="anti")
+
+Duplicate Keys
+--------------
+
+It is common to join two DataFrames on a common column name. Starting in
+version 51.0.0, ``datafusion-python``` will now coalesce on column with identical names by
+default. This reduces problems with ambiguous column selection after joins.
+You can disable this feature by setting the parameter ``coalesce_duplicate_keys``
+to ``False``.
+
+.. ipython:: python
+
+ left = ctx.from_pydict(
+ {
+ "id": [1, 2, 3],
+ "customer": ["Alice", "Bob", "Charlie"],
+ }
+ )
+
+ right = ctx.from_pylist([
+ {"id": 1, "name": "CityCabs"},
+ {"id": 2, "name": "MetroRide"},
+ {"id": 5, "name": "UrbanGo"},
+ ])
+
+ left.join(right, "id", how="inner")
+
+In contrast to the above example, if we wish to get both columns:
+
+.. ipython:: python
+
+ left.join(right, "id", how="inner", coalesce_duplicate_keys=False)
diff --git a/_sources/user-guide/configuration.rst.txt b/_sources/user-guide/configuration.rst.txt
index 5425a040..f8e613cd 100644
--- a/_sources/user-guide/configuration.rst.txt
+++ b/_sources/user-guide/configuration.rst.txt
@@ -15,6 +15,8 @@
.. specific language governing permissions and limitations
.. under the License.
+.. _configuration:
+
Configuration
=============
diff --git a/_sources/user-guide/dataframe/index.rst.txt b/_sources/user-guide/dataframe/index.rst.txt
index 1387db0b..510bcbc6 100644
--- a/_sources/user-guide/dataframe/index.rst.txt
+++ b/_sources/user-guide/dataframe/index.rst.txt
@@ -95,8 +95,9 @@ DataFusion's DataFrame API offers a wide range of operations:
# Select with expressions
df = df.select(column("a") + column("b"), column("a") - column("b"))
- # Filter rows
+ # Filter rows (expressions or SQL strings)
df = df.filter(column("age") > literal(25))
+ df = df.filter("age > 25")
# Add computed columns
df = df.with_column("full_name", column("first_name") + literal(" ") + column("last_name"))
@@ -195,10 +196,121 @@ To materialize the results of your DataFrame operations:
# Display results
df.show() # Print tabular format to console
-
+
# Count rows
count = df.count()
+ # Collect a single column of data as a PyArrow Array
+ arr = df.collect_column("age")
+
+Zero-copy streaming to Arrow-based Python libraries
+---------------------------------------------------
+
+DataFusion DataFrames implement the ``__arrow_c_stream__`` protocol, enabling
+zero-copy, lazy streaming into Arrow-based Python libraries. With the streaming
+protocol, batches are produced on demand.
+
+.. note::
+
+ The protocol is implementation-agnostic and works with any Python library
+ that understands the Arrow C streaming interface (for example, PyArrow
+ or other Arrow-compatible implementations). The sections below provide a
+ short PyArrow-specific example and general guidance for other
+ implementations.
+
+PyArrow
+-------
+
+.. code-block:: python
+
+ import pyarrow as pa
+
+ # Create a PyArrow RecordBatchReader without materializing all batches
+ reader = pa.RecordBatchReader.from_stream(df)
+ for batch in reader:
+ ... # process each batch as it is produced
+
+DataFrames are also iterable, yielding :class:`datafusion.RecordBatch`
+objects lazily so you can loop over results directly without importing
+PyArrow:
+
+.. code-block:: python
+
+ for batch in df:
+ ... # each batch is a ``datafusion.RecordBatch``
+
+Each batch exposes ``to_pyarrow()``, allowing conversion to a PyArrow
+table. ``pa.table(df)`` collects the entire DataFrame eagerly into a
+PyArrow table:
+
+.. code-block:: python
+
+ import pyarrow as pa
+ table = pa.table(df)
+
+Asynchronous iteration is supported as well, allowing integration with
+``asyncio`` event loops:
+
+.. code-block:: python
+
+ async for batch in df:
+ ... # process each batch as it is produced
+
+To work with the stream directly, use ``execute_stream()``, which returns a
+:class:`~datafusion.RecordBatchStream`.
+
+.. code-block:: python
+
+ stream = df.execute_stream()
+ for batch in stream:
+ ...
+
+Execute as Stream
+^^^^^^^^^^^^^^^^^
+
+For finer control over streaming execution, use
+:py:meth:`~datafusion.DataFrame.execute_stream` to obtain a
+:py:class:`datafusion.RecordBatchStream`:
+
+.. code-block:: python
+
+ stream = df.execute_stream()
+ for batch in stream:
+ ... # process each batch as it is produced
+
+.. tip::
+
+ To get a PyArrow reader instead, call
+
+ ``pa.RecordBatchReader.from_stream(df)``.
+
+When partition boundaries are important,
+:py:meth:`~datafusion.DataFrame.execute_stream_partitioned`
+returns an iterable of :py:class:`datafusion.RecordBatchStream` objects, one per
+partition:
+
+.. code-block:: python
+
+ for stream in df.execute_stream_partitioned():
+ for batch in stream:
+ ... # each stream yields RecordBatches
+
+To process partitions concurrently, first collect the streams into a list
+and then poll each one in a separate ``asyncio`` task:
+
+.. code-block:: python
+
+ import asyncio
+
+ async def consume(stream):
+ async for batch in stream:
+ ...
+
+ streams = list(df.execute_stream_partitioned())
+ await asyncio.gather(*(consume(s) for s in streams))
+
+See :doc:`../io/arrow` for additional details on the Arrow interface.
+
HTML Rendering
--------------
diff --git a/_sources/user-guide/io/arrow.rst.txt b/_sources/user-guide/io/arrow.rst.txt
index d571aa99..9196fcea 100644
--- a/_sources/user-guide/io/arrow.rst.txt
+++ b/_sources/user-guide/io/arrow.rst.txt
@@ -60,14 +60,16 @@ Exporting from DataFusion
DataFusion DataFrames implement ``__arrow_c_stream__`` PyCapsule interface, so any
Python library that accepts these can import a DataFusion DataFrame directly.
-.. warning::
- It is important to note that this will cause the DataFrame execution to happen, which may be
- a time consuming task. That is, you will cause a
- :py:func:`datafusion.dataframe.DataFrame.collect` operation call to occur.
+Invoking ``__arrow_c_stream__`` triggers execution of the underlying query, but
+batches are yielded incrementally rather than materialized all at once in memory.
+Consumers can process the stream as it arrives. The stream executes lazily,
+letting downstream readers pull batches on demand.
.. ipython:: python
+ from datafusion import col, lit
+
df = df.select((col("a") * lit(1.5)).alias("c"), lit("df").alias("d"))
pa.table(df)
diff --git a/_sources/user-guide/sql.rst.txt b/_sources/user-guide/sql.rst.txt
index 6fa7f0c6..b4bfb961 100644
--- a/_sources/user-guide/sql.rst.txt
+++ b/_sources/user-guide/sql.rst.txt
@@ -23,17 +23,100 @@ DataFusion also offers a SQL API, read the full reference `here <https://arrow.a
.. ipython:: python
import datafusion
- from datafusion import col
- import pyarrow
+ from datafusion import DataFrame, SessionContext
# create a context
ctx = datafusion.SessionContext()
# register a CSV
- ctx.register_csv('pokemon', 'pokemon.csv')
+ ctx.register_csv("pokemon", "pokemon.csv")
# create a new statement via SQL
df = ctx.sql('SELECT "Attack"+"Defense", "Attack"-"Defense" FROM pokemon')
# collect and convert to pandas DataFrame
- df.to_pandas()
\ No newline at end of file
+ df.to_pandas()
+
+Parameterized queries
+---------------------
+
+In DataFusion-Python 51.0.0 we introduced the ability to pass parameters
+in a SQL query. These are similar in concept to
+`prepared statements <https://datafusion.apache.org/user-guide/sql/prepared_statements.html>`_,
+but allow passing named parameters into a SQL query. Consider this simple
+example.
+
+.. ipython:: python
+
+ def show_attacks(ctx: SessionContext, threshold: int) -> None:
+ ctx.sql(
+ 'SELECT "Name", "Attack" FROM pokemon WHERE "Attack" > $val', val=threshold
+ ).show(num=5)
+ show_attacks(ctx, 75)
+
+When passing parameters like the example above we convert the Python objects
+into their string representation. We also have special case handling
+for :py:class:`~datafusion.dataframe.DataFrame` objects, since they cannot simply
+be turned into string representations for an SQL query. In these cases we
+will register a temporary view in the :py:class:`~datafusion.context.SessionContext`
+using a generated table name.
+
+The formatting for passing string replacement objects is to precede the
+variable name with a single ``$``. This works for all dialects in
+the SQL parser except ``hive`` and ``mysql``. Since these dialects do not
+support named placeholders, we are unable to do this type of replacement.
+We recommend either switching to another dialect or using Python
+f-string style replacement.
+
+.. warning::
+
+ To support DataFrame parameterized queries, your session must support
+ registration of temporary views. The default
+ :py:class:`~datafusion.catalog.CatalogProvider` and
+ :py:class:`~datafusion.catalog.SchemaProvider` do have this capability.
+ If you have implemented custom providers, it is important that temporary
+ views do not persist across :py:class:`~datafusion.context.SessionContext`
+ or you may get unintended consequences.
+
+The following example shows passing in both a :py:class:`~datafusion.dataframe.DataFrame`
+object as well as a Python object to be used in parameterized replacement.
+
+.. ipython:: python
+
+ def show_column(
+ ctx: SessionContext, column: str, df: DataFrame, threshold: int
+ ) -> None:
+ ctx.sql(
+ 'SELECT "Name", $col FROM $df WHERE $col > $val',
+ col=column,
+ df=df,
+ val=threshold,
+ ).show(num=5)
+ df = ctx.table("pokemon")
+ show_column(ctx, '"Defense"', df, 75)
+
+The approach implemented for conversion of variables into a SQL query
+relies on string conversion. This has the potential for data loss,
+specifically for cases like floating point numbers. If you need to pass
+variables into a parameterized query and it is important to maintain the
+original value without conversion to a string, then you can use the
+optional parameter ``param_values`` to specify these. This parameter
+expects a dictionary mapping from the parameter name to a Python
+object. Those objects will be cast into a
+`PyArrow Scalar Value <https://arrow.apache.org/docs/python/generated/pyarrow.Scalar.html>`_.
+
+Using ``param_values`` will rely on the SQL dialect you have configured
+for your session. This can be set using the :ref:`configuration options <configuration>`
+of your :py:class:`~datafusion.context.SessionContext`. Similar to how
+`prepared statements <https://datafusion.apache.org/user-guide/sql/prepared_statements.html>`_
+work, these parameters are limited to places where you would pass in a
+scalar value, such as a comparison.
+
+.. ipython:: python
+
+ def param_attacks(ctx: SessionContext, threshold: int) -> None:
+ ctx.sql(
+ 'SELECT "Name", "Attack" FROM pokemon WHERE "Attack" > $val',
+ param_values={"val": threshold},
+ ).show(num=5)
+ param_attacks(ctx, 75)
diff --git a/autoapi/datafusion/context/index.html b/autoapi/datafusion/context/index.html
index cfd76c86..f8aefac9 100644
--- a/autoapi/datafusion/context/index.html
+++ b/autoapi/datafusion/context/index.html
@@ -2282,8 +2282,11 @@ this context.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.sql">
-<span class="sig-name descname"><span class="pre">sql</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">query</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">options</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference i [...]
+<span class="sig-name descname"><span class="pre">sql</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">query</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">options</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference i [...]
<dd><p>Create a <code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code> from SQL query text.</p>
+<p>See the online documentation for a description of how to perform
+parameterized substitution via either the <code class="docutils literal notranslate"><span class="pre">param_values</span></code> option
+or passing in <code class="docutils literal notranslate"><span class="pre">named_params</span></code>.</p>
<p>Note: This API implements DDL statements such as <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and
<code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">VIEW</span></code> and DML statements such as <code class="docutils literal notranslate"><span class="pre">INSERT</span> <span class="pre">INTO</span></code> with in-memory
default implementation.See
@@ -2293,6 +2296,9 @@ default implementation.See
<dd class="field-odd"><ul class="simple">
<li><p><strong>query</strong> – SQL query text.</p></li>
<li><p><strong>options</strong> – If provided, the query will be validated against these options.</p></li>
+<li><p><strong>param_values</strong> – Provides substitution of scalar values in the query
+after parsing.</p></li>
+<li><p><strong>named_params</strong> – Provides string or DataFrame substitution in the query string.</p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
@@ -2303,7 +2309,7 @@ default implementation.See
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.sql_with_options">
-<span class="sig-name descname"><span class="pre">sql_with_options</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">query</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">options</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class [...]
+<span class="sig-name descname"><span class="pre">sql_with_options</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">query</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">options</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class [...]
<dd><p>Create a <a class="reference internal" href="../dataframe/index.html#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a> from SQL query text.</p>
<p>This function will first validate that the query is allowed by the
provided options.</p>
@@ -2312,6 +2318,9 @@ provided options.</p>
<dd class="field-odd"><ul class="simple">
<li><p><strong>query</strong> – SQL query text.</p></li>
<li><p><strong>options</strong> – SQL options.</p></li>
+<li><p><strong>param_values</strong> – Provides substitution of scalar values in the query
+after parsing.</p></li>
+<li><p><strong>named_params</strong> – Provides string or DataFrame substitution in the query string.</p></li>
</ul>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
diff --git a/autoapi/datafusion/dataframe/index.html b/autoapi/datafusion/dataframe/index.html
index b302fbec..15d90ff7 100644
--- a/autoapi/datafusion/dataframe/index.html
+++ b/autoapi/datafusion/dataframe/index.html
@@ -518,6 +518,15 @@
</code>
</a>
<ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.__aiter__">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ DataFrame.__aiter__()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.__arrow_c_stream__">
<code class="docutils literal notranslate">
@@ -536,6 +545,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.__iter__">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ DataFrame.__iter__()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.__repr__">
<code class="docutils literal notranslate">
@@ -590,6 +608,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.collect_column">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ DataFrame.collect_column()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.collect_partitioned">
<code class="docutils literal notranslate">
@@ -833,6 +860,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.select_exprs">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ DataFrame.select_exprs()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.dataframe.DataFrame.show">
<code class="docutils literal notranslate">
@@ -1475,32 +1511,56 @@
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame">
<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.dataframe.</span></span><span class="sig-name descname"><span class="pre">DataFrame</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">df</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">datafusion._internal.DataFrame</spa [...]
<dd><p>Two dimensional table representation of data.</p>
+<p>DataFrame objects are iterable; iterating over a DataFrame yields
+<a class="reference internal" href="../index.html#datafusion.RecordBatch" title="datafusion.RecordBatch"><code class="xref py py-class docutils literal notranslate"><span class="pre">datafusion.RecordBatch</span></code></a> instances lazily.</p>
<p>See <a class="reference internal" href="../../../user-guide/basics.html#user-guide-concepts"><span class="std std-ref">Concepts</span></a> in the online documentation for more information.</p>
<p>This constructor is not to be used by the end user.</p>
<p>See <a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a> for methods to
create a <a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a>.</p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.__aiter__">
+<span class="sig-name descname"><span class="pre">__aiter__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">collections.abc.AsyncIterator</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="../record_batch/index.html#datafusion.record_batch.RecordBatch" title="datafusion.record_batch.RecordBatch"><s [...]
+<dd><p>Return an async iterator over this DataFrame’s record batches.</p>
+<p>We’re using __aiter__ because we support Python < 3.10 where aiter() is not
+available.</p>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.__arrow_c_stream__">
<span class="sig-name descname"><span class="pre">__arrow_c_stream__</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">requested_schema</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">object</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o" [...]
-<dd><p>Export an Arrow PyCapsule Stream.</p>
-<p>This will execute and collect the DataFrame. We will attempt to respect the
-requested schema, but only trivial transformations will be applied such as only
-returning the fields listed in the requested schema if their data types match
-those in the DataFrame.</p>
+<dd><p>Export the DataFrame as an Arrow C Stream.</p>
+<p>The DataFrame is executed using DataFusion’s streaming APIs and exposed via
+Arrow’s C Stream interface. Record batches are produced incrementally, so the
+full result set is never materialized in memory.</p>
+<p>When <code class="docutils literal notranslate"><span class="pre">requested_schema</span></code> is provided, DataFusion applies only simple
+projections such as selecting a subset of existing columns or reordering
+them. Column renaming, computed expressions, or type coercion are not
+supported through this interface.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
-<dd class="field-odd"><p><strong>requested_schema</strong> – Attempt to provide the DataFrame using this schema.</p>
+<dd class="field-odd"><p><strong>requested_schema</strong> – Either a <code class="xref py py-class docutils literal notranslate"><span class="pre">pyarrow.Schema</span></code> or an Arrow C
+Schema capsule (<code class="docutils literal notranslate"><span class="pre">PyCapsule</span></code>) produced by
+<code class="docutils literal notranslate"><span class="pre">schema._export_to_c_capsule()</span></code>. The DataFrame will attempt to
+align its output with the fields and order specified by this schema.</p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
-<dd class="field-even"><p>Arrow PyCapsule object.</p>
+<dd class="field-even"><p>Arrow <code class="docutils literal notranslate"><span class="pre">PyCapsule</span></code> object representing an <code class="docutils literal notranslate"><span class="pre">ArrowArrayStream</span></code>.</p>
</dd>
</dl>
+<p>For practical usage patterns, see the Apache Arrow streaming
+documentation: <a class="reference external" href="https://arrow.apache.org/docs/python/ipc.html#streaming">https://arrow.apache.org/docs/python/ipc.html#streaming</a>.</p>
+<p>For details on DataFusion’s Arrow integration and DataFrame streaming,
+see the user guide (user-guide/io/arrow and user-guide/dataframe/index).</p>
+<p class="rubric">Notes</p>
+<p>The Arrow C Data Interface PyCapsule details are documented by Apache
+Arrow and can be found at:
+<a class="reference external" href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html</a></p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.__getitem__">
<span class="sig-name descname"><span class="pre">__getitem__</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">key</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">st [...]
-<dd><p>Return a new :py:class`DataFrame` with the specified column or columns.</p>
+<dd><p>Return a new <a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a> with the specified column or columns.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><p><strong>key</strong> – Column name or list of column names to select.</p>
@@ -1511,6 +1571,12 @@ those in the DataFrame.</p>
</dl>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.__iter__">
+<span class="sig-name descname"><span class="pre">__iter__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">collections.abc.Iterator</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="../record_batch/index.html#datafusion.record_batch.RecordBatch" title="datafusion.record_batch.RecordBatch"><span cl [...]
+<dd><p>Return an iterator over this DataFrame’s record batches.</p>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.__repr__">
<span class="sig-name descname"><span class="pre">__repr__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">str</span></span></span><a class="headerlink" href="#datafusion.dataframe.DataFrame.__repr__" title="Link to this definition">¶</a></dt>
@@ -1583,6 +1649,12 @@ computation.</p>
</dl>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.collect_column">
+<span class="sig-name descname"><span class="pre">collect_column</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">column_name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">pyarrow.Array</s [...]
+<dd><p>Executes this <a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a> for a single column.</p>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.collect_partitioned">
<span class="sig-name descname"><span class="pre">collect_partitioned</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">pyarrow.RecordBatch</span><span class="p"><span class="pre">]</sp [...]
@@ -1763,23 +1835,24 @@ and <code class="docutils literal notranslate"><span class="pre">"column_na
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.filter">
-<span class="sig-name descname"><span class="pre">filter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">predicates</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a></span></em><span [...]
+<span class="sig-name descname"><span class="pre">filter</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">predicates</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a><span class="w"> [...]
<dd><p>Return a DataFrame for which <code class="docutils literal notranslate"><span class="pre">predicate</span></code> evaluates to <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>
<p>Rows for which <code class="docutils literal notranslate"><span class="pre">predicate</span></code> evaluates to <code class="docutils literal notranslate"><span class="pre">False</span></code> or <code class="docutils literal notranslate"><span class="pre">None</span></code> are filtered
out. If more than one predicate is provided, these predicates will be
-combined as a logical AND. Each <code class="docutils literal notranslate"><span class="pre">predicate</span></code> must be an
+combined as a logical AND. Each <code class="docutils literal notranslate"><span class="pre">predicate</span></code> can be an
<a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code></a> created using helper functions such as
-<a class="reference internal" href="../index.html#datafusion.col" title="datafusion.col"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.col()</span></code></a> or <a class="reference internal" href="../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.lit()</span></code></a>.
-If more complex logic is required, see the logical operations in
-<a class="reference internal" href="../functions/index.html#module-datafusion.functions" title="datafusion.functions"><code class="xref py py-mod docutils literal notranslate"><span class="pre">functions</span></code></a>.</p>
+<a class="reference internal" href="../index.html#datafusion.col" title="datafusion.col"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.col()</span></code></a> or <a class="reference internal" href="../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.lit()</span></code></a>, or a SQL expression string
+that will be parsed against the DataFrame schema. If more complex logic is
+required, see the logical operations in <a class="reference internal" href="../functions/index.html#module-datafusion.functions" title="datafusion.functions"><code class="xref py py-mod docutils literal notranslate"><span class="pre">functions</span></code></a>.</p>
<p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">col</span><span class="p">,</span> <span class="n">lit</span>
<span class="n">df</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">></span> <span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
+<span class="n">df</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s2">"a > 1"</span><span class="p">)</span>
</pre></div>
</div>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
-<dd class="field-odd"><p><strong>predicates</strong> – Predicate expression(s) to filter the DataFrame.</p>
+<dd class="field-odd"><p><strong>predicates</strong> – Predicate expression(s) or SQL strings to filter the DataFrame.</p>
</dd>
<dt class="field-even">Returns<span class="colon">:</span></dt>
<dd class="field-even"><p>DataFrame after filtering.</p>
@@ -1818,7 +1891,7 @@ If more complex logic is required, see the logical operations in
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.into_view">
-<span class="sig-name descname"><span class="pre">into_view</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../catalog/index.html#datafusion.catalog.Table" title="datafusion.catalog.Table"><span class="pre">datafusion.catalog.Table</span></a></span></span><a class="headerlink" href="#datafusion.dataframe.DataFrame.into_ [...]
+<span class="sig-name descname"><span class="pre">into_view</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">temporary</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">False</span></span></em><span class="sig [...]
<dd><p>Convert <code class="docutils literal notranslate"><span class="pre">DataFrame</span></code> into a <a class="reference internal" href="../index.html#datafusion.Table" title="datafusion.Table"><code class="xref py py-class docutils literal notranslate"><span class="pre">Table</span></code></a>.</p>
<p class="rubric">Examples</p>
<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">SessionContext</span>
@@ -1834,11 +1907,11 @@ If more complex logic is required, see the logical operations in
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.join">
-<span class="sig-name descname"><span class="pre">join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">right</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><span class="pre">DataFrame</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">on</span></s [...]
+<span class="sig-name descname"><span class="pre">join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">right</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><span class="pre">DataFrame</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">on</span></s [...]
<dt class="sig sig-object py">
-<span class="sig-name descname"><span class="pre">join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">right</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><span class="pre">DataFrame</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">on</span></s [...]
+<span class="sig-name descname"><span class="pre">join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">right</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><span class="pre">DataFrame</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">on</span></s [...]
<dt class="sig sig-object py">
-<span class="sig-name descname"><span class="pre">join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">right</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><span class="pre">DataFrame</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">on</span></s [...]
+<span class="sig-name descname"><span class="pre">join</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">right</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><span class="pre">DataFrame</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">on</span></s [...]
<dd><p>Join this <a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a> with another <a class="reference internal" href="#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a>.</p>
<p><cite>on</cite> has to be provided or both <cite>left_on</cite> and <cite>right_on</cite> in conjunction.</p>
<dl class="field-list simple">
@@ -1850,6 +1923,9 @@ If more complex logic is required, see the logical operations in
“right”, “full”, “semi”, “anti”.</p></li>
<li><p><strong>left_on</strong> – Join column of the left dataframe.</p></li>
<li><p><strong>right_on</strong> – Join column of the right dataframe.</p></li>
+<li><p><strong>coalesce_duplicate_keys</strong> – When True, coalesce the columns
+from the right DataFrame and left DataFrame
+that have identical names in the <code class="docutils literal notranslate"><span class="pre">on</span></code> fields.</p></li>
<li><p><strong>join_keys</strong> – Tuple of two lists of column names to join on. [Deprecated]</p></li>
</ul>
</dd>
@@ -1966,12 +2042,13 @@ in-equality predicates. Equality predicates are correctly optimized.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.repartition_by_hash">
-<span class="sig-name descname"><span class="pre">repartition_by_hash</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">exprs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a></span></ [...]
+<span class="sig-name descname"><span class="pre">repartition_by_hash</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">exprs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a><span cla [...]
<dd><p>Repartition a DataFrame using a hash partitioning scheme.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
-<li><p><strong>exprs</strong> – Expressions to evaluate and perform hashing on.</p></li>
+<li><p><strong>exprs</strong> – Expressions or a SQL expression string to evaluate
+and perform hashing on.</p></li>
<li><p><strong>num</strong> – Number of partitions to repartition the DataFrame into.</p></li>
</ul>
</dd>
@@ -2027,6 +2104,19 @@ column <code class="docutils literal notranslate"><span class="pre">a</span></co
</dl>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.select_exprs">
+<span class="sig-name descname"><span class="pre">select_exprs</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-type [...]
+<dd><p>Project arbitrary list of expression strings into a new DataFrame.</p>
+<p>This method will parse string expressions into logical plan expressions.
+The output DataFrame has one column for each expression.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns<span class="colon">:</span></dt>
+<dd class="field-odd"><p>DataFrame only containing the specified columns.</p>
+</dd>
+</dl>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.show">
<span class="sig-name descname"><span class="pre">show</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">num</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">20</span></span></em><span class="sig-paren">)</span [...]
@@ -2127,7 +2217,7 @@ determined of the dataframe. This is done by collecting it.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.transform">
-<span class="sig-name descname"><span class="pre">transform</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference internal" href=" [...]
+<span class="sig-name descname"><span class="pre">transform</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><a class="reference [...]
<dd><p>Apply a function to the current DataFrame which returns another DataFrame.</p>
<p>This is useful for chaining together multiple functions. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">add_3</span><span class="p">(</span><span class="n">df</span><span class="p">:</span> <span class="n">DataFrame</span><span class="p">)</span> <span class="o">-></span> <span class="n">DataFrame</span><span class="p">:</span>
@@ -2209,10 +2299,11 @@ returned.</p></li>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.with_column">
-<span class="sig-name descname"><span class="pre">with_column</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="referen [...]
+<span class="sig-name descname"><span class="pre">with_column</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">expr</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="referen [...]
<dd><p>Add an additional column to the DataFrame.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">expr</span></code> must be an <a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code></a> constructed with
-<a class="reference internal" href="../index.html#datafusion.col" title="datafusion.col"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.col()</span></code></a> or <a class="reference internal" href="../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.lit()</span></code></a>.</p>
+<a class="reference internal" href="../index.html#datafusion.col" title="datafusion.col"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.col()</span></code></a> or <a class="reference internal" href="../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.lit()</span></code></a>, or a SQL expression
+string that will be parsed against the DataFrame schema.</p>
<p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">col</span><span class="p">,</span> <span class="n">lit</span>
<span class="n">df</span><span class="o">.</span><span class="n">with_column</span><span class="p">(</span><span class="s2">"b"</span><span class="p">,</span> <span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
@@ -2253,11 +2344,12 @@ into one the following symbols (” or ‘ or `).</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe.DataFrame.with_columns">
-<span class="sig-name descname"><span class="pre">with_columns</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">exprs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a><span class="w"> [...]
+<span class="sig-name descname"><span class="pre">with_columns</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">exprs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><span class="pre">datafusion.expr.Expr</span></a><span class="w"> [...]
<dd><p>Add columns to the DataFrame.</p>
-<p>By passing expressions, iterables of expressions, or named expressions.
+<p>By passing expressions, iterables of expressions, string SQL expressions,
+or named expressions.
All expressions must be <a class="reference internal" href="../expr/index.html#datafusion.expr.Expr" title="datafusion.expr.Expr"><code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code></a> objects created via
-<a class="reference internal" href="../index.html#datafusion.col" title="datafusion.col"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.col()</span></code></a> or <a class="reference internal" href="../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.lit()</span></code></a>.
+<a class="reference internal" href="../index.html#datafusion.col" title="datafusion.col"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.col()</span></code></a> or <a class="reference internal" href="../index.html#datafusion.lit" title="datafusion.lit"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.lit()</span></code></a>, or SQL expression strings.
To pass named expressions use the form <code class="docutils literal notranslate"><span class="pre">name=Expr</span></code>.</p>
<p>Example usage: The following will add 4 columns labeled <code class="docutils literal notranslate"><span class="pre">a</span></code>, <code class="docutils literal notranslate"><span class="pre">b</span></code>, <code class="docutils literal notranslate"><span class="pre">c</span></code>,
and <code class="docutils literal notranslate"><span class="pre">d</span></code>:</p>
@@ -2267,12 +2359,21 @@ and <code class="docutils literal notranslate"><span class="pre">d</span></code>
<span class="p">[</span><span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span> <span class="n">col</span><span class="p">(</span><span class="s2">"y"</span><span class="p">)</span><span class="o">.</span><span class="n">alias</span><span class="p">(</span><span class="s2">"c"</span><s [...]
<span class="n">d</span><span class="o">=</span><span class="n">lit</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="p">)</span>
+
+<span class="n">Equivalent</span> <span class="n">example</span> <span class="n">using</span> <span class="n">just</span> <span class="n">SQL</span> <span class="n">strings</span><span class="p">:</span>
+
+<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">with_columns</span><span class="p">(</span>
+ <span class="s2">"x as a"</span><span class="p">,</span>
+ <span class="p">[</span><span class="s2">"1 as b"</span><span class="p">,</span> <span class="s2">"y as c"</span><span class="p">],</span>
+ <span class="n">d</span><span class="o">=</span><span class="s2">"3"</span>
+<span class="p">)</span>
</pre></div>
</div>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
<dd class="field-odd"><ul class="simple">
-<li><p><strong>exprs</strong> – Either a single expression or an iterable of expressions to add.</p></li>
+<li><p><strong>exprs</strong> – Either a single expression, an iterable of expressions to add or
+SQL expression strings.</p></li>
<li><p><strong>named_exprs</strong> – Named expressions in the form of <code class="docutils literal notranslate"><span class="pre">name=expr</span></code></p></li>
</ul>
</dd>
diff --git a/autoapi/datafusion/dataframe_formatter/index.html b/autoapi/datafusion/dataframe_formatter/index.html
index 2fa77cd1..615c37da 100644
--- a/autoapi/datafusion/dataframe_formatter/index.html
+++ b/autoapi/datafusion/dataframe_formatter/index.html
@@ -1181,7 +1181,7 @@ a formatted string</p></li>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe_formatter.DataFrameHtmlFormatter.set_custom_cell_builder">
-<span class="sig-name descname"><span class="pre">set_custom_cell_builder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">builder</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span></span [...]
+<span class="sig-name descname"><span class="pre">set_custom_cell_builder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">builder</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre [...]
<dd><p>Set a custom cell builder function.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -1192,7 +1192,7 @@ a formatted string</p></li>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.dataframe_formatter.DataFrameHtmlFormatter.set_custom_header_builder">
-<span class="sig-name descname"><span class="pre">set_custom_header_builder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">builder</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></sp [...]
+<span class="sig-name descname"><span class="pre">set_custom_header_builder</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">builder</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="p [...]
<dd><p>Set a custom header builder function.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
@@ -1203,12 +1203,12 @@ a formatted string</p></li>
<dl class="py attribute">
<dt class="sig sig-object py" id="datafusion.dataframe_formatter.DataFrameHtmlFormatter._custom_cell_builder">
-<span class="sig-name descname"><span class="pre">_custom_cell_builder</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">int</span><span class="p"><span class="pre">,</span></span><span [...]
+<span class="sig-name descname"><span class="pre">_custom_cell_builder</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">int</span><span class="p"><span class="pre">,</s [...]
<dd></dd></dl>
<dl class="py attribute">
<dt class="sig sig-object py" id="datafusion.dataframe_formatter.DataFrameHtmlFormatter._custom_header_builder">
-<span class="sig-name descname"><span class="pre">_custom_header_builder</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">str</span><sp [...]
+<span class="sig-name descname"><span class="pre">_custom_header_builder</span></span><em class="property"><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pr [...]
<dd></dd></dl>
<dl class="py attribute">
diff --git a/autoapi/datafusion/expr/index.html b/autoapi/datafusion/expr/index.html
index 69206736..e55ea872 100644
--- a/autoapi/datafusion/expr/index.html
+++ b/autoapi/datafusion/expr/index.html
@@ -3518,7 +3518,7 @@ type of expression, an error will be generated when <code class="docutils litera
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.expr.Expr.python_value">
<span class="sig-name descname"><span class="pre">python_value</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">Any</span></span></span><a class="headerlink" href="#datafusion.expr.Expr.python_value" title="Link to this definition">¶</a></dt>
-<dd><p>Extracts the Expr value into a PyObject.</p>
+<dd><p>Extracts the Expr value into <cite>Any</cite>.</p>
<p>This is only valid for literal expressions.</p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
@@ -3948,7 +3948,7 @@ higher level APIs consistently require explicit <a class="reference internal" hr
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.expr.ensure_expr_list">
-<span class="sig-prename descclassname"><span class="pre">datafusion.expr.</span></span><span class="sig-name descname"><span class="pre">ensure_expr_list</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">exprs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Iterable</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#datafusio [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.expr.</span></span><span class="sig-name descname"><span class="pre">ensure_expr_list</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">exprs</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Iterable</span><span class="p"><span class="pre">[</span></span><a class="reference internal" [...]
<dd><p>Flatten an iterable of expressions, validating each via <code class="docutils literal notranslate"><span class="pre">ensure_expr</span></code>.</p>
<dl class="field-list simple">
<dt class="field-odd">Parameters<span class="colon">:</span></dt>
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index fab8b510..d306bc46 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -2252,6 +2252,15 @@
</code>
</a>
<ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.RecordBatch.__arrow_c_array__">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ RecordBatch.__arrow_c_array__()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.RecordBatch.to_pyarrow">
<code class="docutils literal notranslate">
@@ -3274,7 +3283,9 @@ See <a class="reference external" href="https://datafusion.apache.org/python">ht
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.AggregateUDF">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">AggregateUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-p [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">AggregateUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-p [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">AggregateUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-p [...]
<dd><p>Class for performing scalar user-defined functions (UDF).</p>
<p>Aggregate UDFs operate on a group of rows and return a single value. See
also <a class="reference internal" href="#datafusion.ScalarUDF" title="datafusion.ScalarUDF"><code class="xref py py-class docutils literal notranslate"><span class="pre">ScalarUDF</span></code></a> for operating on a row by row basis.</p>
@@ -3297,7 +3308,7 @@ occur during the evaluation of the dataframe.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.AggregateUDF.from_pycapsule">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="user_defined/index.html#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="user_defined/index.html#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined [...]
<dd><p>Create an Aggregate UDF from AggregateUDF PyCapsule object.</p>
<p>This function will instantiate a Aggregate UDF that uses a DataFusion
AggregateUDF that is exported via the FFI bindings.</p>
@@ -3305,9 +3316,13 @@ AggregateUDF that is exported via the FFI bindings.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.AggregateUDF.udaf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span> [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="user_defined/index.html#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.Aggregat [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="p"> [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">_typeshed.CapsuleType</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">& [...]
<dd><p>Create a new User-Defined Aggregate Function (UDAF).</p>
<p>This class allows you to define an aggregate function that can be used in
data aggregation or window function calls.</p>
@@ -4143,7 +4158,7 @@ type of expression, an error will be generated when <code class="docutils litera
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.Expr.python_value">
<span class="sig-name descname"><span class="pre">python_value</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">Any</span></span></span><a class="headerlink" href="#datafusion.Expr.python_value" title="Link to this definition">¶</a></dt>
-<dd><p>Extracts the Expr value into a PyObject.</p>
+<dd><p>Extracts the Expr value into <cite>Any</cite>.</p>
<p>This is only valid for literal expressions.</p>
<dl class="field-list simple">
<dt class="field-odd">Returns<span class="colon">:</span></dt>
@@ -4749,6 +4764,25 @@ here.</p></li>
<dd><p>This class is essentially a wrapper for <code class="xref py py-class docutils literal notranslate"><span class="pre">pa.RecordBatch</span></code>.</p>
<p>This constructor is generally not called by the end user.</p>
<p>See the <a class="reference internal" href="#datafusion.RecordBatchStream" title="datafusion.RecordBatchStream"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatchStream</span></code></a> iterator for generating this class.</p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.RecordBatch.__arrow_c_array__">
+<span class="sig-name descname"><span class="pre">__arrow_c_array__</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">requested_schema</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">object</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"> [...]
+<dd><p>Export the record batch via the Arrow C Data Interface.</p>
+<p>This allows zero-copy interchange with libraries that support the
+<a class="reference external" href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule interface</a>.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><p><strong>requested_schema</strong> – Attempt to provide the record batch using this
+schema. Only straightforward projections such as column
+selection or reordering are applied.</p>
+</dd>
+<dt class="field-even">Returns<span class="colon">:</span></dt>
+<dd class="field-even"><p>Two Arrow PyCapsule objects representing the <code class="docutils literal notranslate"><span class="pre">ArrowArray</span></code> and
+<code class="docutils literal notranslate"><span class="pre">ArrowSchema</span></code>.</p>
+</dd>
+</dl>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.RecordBatch.to_pyarrow">
<span class="sig-name descname"><span class="pre">to_pyarrow</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">pyarrow.RecordBatch</span></span></span><a class="headerlink" href="#datafusion.RecordBatch.to_pyarrow" title="Link to this definition">¶</a></dt>
@@ -4772,25 +4806,25 @@ here.</p></li>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.RecordBatchStream.__aiter__">
<span class="sig-name descname"><span class="pre">__aiter__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">typing_extensions.Self</span></span></span><a class="headerlink" href="#datafusion.RecordBatchStream.__aiter__" title="Link to this definition">¶</a></dt>
-<dd><p>Async iterator function.</p>
+<dd><p>Return an asynchronous iterator over record batches.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.RecordBatchStream.__anext__">
<em class="property"><span class="pre">async</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">__anext__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.RecordBatch" title="datafusion.RecordBatch"><span class="pre">RecordBatch</span></a></span></span><a class="headerlink" h [...]
-<dd><p>Async iterator function.</p>
+<dd><p>Return the next <a class="reference internal" href="#datafusion.RecordBatch" title="datafusion.RecordBatch"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatch</span></code></a> in the stream asynchronously.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.RecordBatchStream.__iter__">
<span class="sig-name descname"><span class="pre">__iter__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">typing_extensions.Self</span></span></span><a class="headerlink" href="#datafusion.RecordBatchStream.__iter__" title="Link to this definition">¶</a></dt>
-<dd><p>Iterator function.</p>
+<dd><p>Return an iterator over record batches.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.RecordBatchStream.__next__">
<span class="sig-name descname"><span class="pre">__next__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.RecordBatch" title="datafusion.RecordBatch"><span class="pre">RecordBatch</span></a></span></span><a class="headerlink" href="#datafusion.RecordBatchStream.__next__" title="Link to this definition">¶</a></dt>
-<dd><p>Iterator function.</p>
+<dd><p>Return the next <a class="reference internal" href="#datafusion.RecordBatch" title="datafusion.RecordBatch"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatch</span></code></a> in the stream.</p>
</dd></dl>
<dl class="py method">
@@ -5013,7 +5047,7 @@ multiple spillable operators that all will spill.</p>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.ScalarUDF">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">ScalarUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-para [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">ScalarUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-para [...]
<dd><p>Class for performing scalar user-defined functions (UDF).</p>
<p>Scalar UDFs operate on a row by row basis. See also <a class="reference internal" href="#datafusion.AggregateUDF" title="datafusion.AggregateUDF"><code class="xref py py-class docutils literal notranslate"><span class="pre">AggregateUDF</span></code></a> for
operating on a group of rows.</p>
@@ -5043,9 +5077,9 @@ ScalarUDF that is exported via the FFI bindings.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.ScalarUDF.udf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">pyarrow.DataType</span><span class="p"><span c [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">pyarrow.DataType</span><span class="p"><span c [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><span class="pre"> [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><s [...]
<dt class="sig sig-object py">
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="user_defined/index.html#datafusion.user_defined.ScalarUDFExportable" title="datafusion.user_defined.ScalarUDFExpo [...]
<dd><p>Create a new User-Defined Function (UDF).</p>
@@ -5361,7 +5395,7 @@ interface (advanced)</p></li>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.TableFunction">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">TableFunction</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig- [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">TableFunction</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig- [...]
<dd><p>Class for performing user-defined table functions (UDTF).</p>
<p>Table functions generate new table providers based on the
input expressions.</p>
@@ -5382,21 +5416,21 @@ descriptions.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.TableFunction._create_table_udf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><spa [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><spa [...]
<dd><p>Create a TableFunction instance from function arguments.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.TableFunction._create_table_udf_decorator">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span> [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span> [...]
<dd><p>Create a decorator for a WindowUDF.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.TableFunction.udtf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <spa [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <spa [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="p">< [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span>< [...]
<dd><p>Create a new User-Defined Table Function (UDTF).</p>
</dd></dl>
@@ -5458,7 +5492,7 @@ parameter must be set.</p></li>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.WindowUDF">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">WindowUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-para [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">WindowUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-para [...]
<dd><p>Class for performing window user-defined functions (UDF).</p>
<p>Window UDFs operate on a partition of rows. See
also <a class="reference internal" href="#datafusion.ScalarUDF" title="datafusion.ScalarUDF"><code class="xref py py-class docutils literal notranslate"><span class="pre">ScalarUDF</span></code></a> for operating on a row by row basis.</p>
@@ -5481,19 +5515,19 @@ occur during the evaluation of the dataframe.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.WindowUDF._create_window_udf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[< [...]
<dd><p>Create a WindowUDF instance from function arguments.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.WindowUDF._create_window_udf_decorator">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><sp [...]
<dd><p>Create a decorator for a WindowUDF.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.WindowUDF._get_default_name">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_get_default_name</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&# [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_get_default_name</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig [...]
<dd><p>Get the default name for a function based on its attributes.</p>
</dd></dl>
@@ -5513,9 +5547,9 @@ WindowUDF that is exported via the FFI bindings.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.WindowUDF.udwf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="p">< [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span>< [...]
<dd><p>Create a new User-Defined Window Function (UDWF).</p>
<p>This class can be used both as either a function or a decorator.</p>
<dl class="simple">
@@ -5607,13 +5641,13 @@ max_width, max_height, enable_cell_expansion, etc.</p>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.lit">
-<span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">lit</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="expr/index.html#datafusion.expr.Expr" title="datafusion. [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">lit</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Any</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <s [...]
<dd><p>Create a literal expression.</p>
</dd></dl>
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.literal">
-<span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">literal</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="expr/index.html#datafusion.expr.Expr" title="datafus [...]
+<span class="sig-prename descclassname"><span class="pre">datafusion.</span></span><span class="sig-name descname"><span class="pre">literal</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Any</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span [...]
<dd><p>Create a literal expression.</p>
</dd></dl>
diff --git a/autoapi/datafusion/record_batch/index.html b/autoapi/datafusion/record_batch/index.html
index ec09d72c..e36e3aa6 100644
--- a/autoapi/datafusion/record_batch/index.html
+++ b/autoapi/datafusion/record_batch/index.html
@@ -426,6 +426,15 @@
</code>
</a>
<ul class="nav section-nav flex-column">
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.record_batch.RecordBatch.__arrow_c_array__">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ RecordBatch.__arrow_c_array__()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.record_batch.RecordBatch.to_pyarrow">
<code class="docutils literal notranslate">
@@ -560,6 +569,25 @@
<dd><p>This class is essentially a wrapper for <code class="xref py py-class docutils literal notranslate"><span class="pre">pa.RecordBatch</span></code>.</p>
<p>This constructor is generally not called by the end user.</p>
<p>See the <a class="reference internal" href="#datafusion.record_batch.RecordBatchStream" title="datafusion.record_batch.RecordBatchStream"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatchStream</span></code></a> iterator for generating this class.</p>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.record_batch.RecordBatch.__arrow_c_array__">
+<span class="sig-name descname"><span class="pre">__arrow_c_array__</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">requested_schema</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">object</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"> [...]
+<dd><p>Export the record batch via the Arrow C Data Interface.</p>
+<p>This allows zero-copy interchange with libraries that support the
+<a class="reference external" href="https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html">Arrow PyCapsule interface</a>.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><p><strong>requested_schema</strong> – Attempt to provide the record batch using this
+schema. Only straightforward projections such as column
+selection or reordering are applied.</p>
+</dd>
+<dt class="field-even">Returns<span class="colon">:</span></dt>
+<dd class="field-even"><p>Two Arrow PyCapsule objects representing the <code class="docutils literal notranslate"><span class="pre">ArrowArray</span></code> and
+<code class="docutils literal notranslate"><span class="pre">ArrowSchema</span></code>.</p>
+</dd>
+</dl>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.record_batch.RecordBatch.to_pyarrow">
<span class="sig-name descname"><span class="pre">to_pyarrow</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">pyarrow.RecordBatch</span></span></span><a class="headerlink" href="#datafusion.record_batch.RecordBatch.to_pyarrow" title="Link to this definition">¶</a></dt>
@@ -583,25 +611,25 @@
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.record_batch.RecordBatchStream.__aiter__">
<span class="sig-name descname"><span class="pre">__aiter__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">typing_extensions.Self</span></span></span><a class="headerlink" href="#datafusion.record_batch.RecordBatchStream.__aiter__" title="Link to this definition">¶</a></dt>
-<dd><p>Async iterator function.</p>
+<dd><p>Return an asynchronous iterator over record batches.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.record_batch.RecordBatchStream.__anext__">
<em class="property"><span class="pre">async</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">__anext__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.record_batch.RecordBatch" title="datafusion.record_batch.RecordBatch"><span class="pre">RecordBatch</span></a></span></sp [...]
-<dd><p>Async iterator function.</p>
+<dd><p>Return the next <a class="reference internal" href="#datafusion.record_batch.RecordBatch" title="datafusion.record_batch.RecordBatch"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatch</span></code></a> in the stream asynchronously.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.record_batch.RecordBatchStream.__iter__">
<span class="sig-name descname"><span class="pre">__iter__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">typing_extensions.Self</span></span></span><a class="headerlink" href="#datafusion.record_batch.RecordBatchStream.__iter__" title="Link to this definition">¶</a></dt>
-<dd><p>Iterator function.</p>
+<dd><p>Return an iterator over record batches.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.record_batch.RecordBatchStream.__next__">
<span class="sig-name descname"><span class="pre">__next__</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="#datafusion.record_batch.RecordBatch" title="datafusion.record_batch.RecordBatch"><span class="pre">RecordBatch</span></a></span></span><a class="headerlink" href="#datafusion.record_batch.RecordBatchStream.__next_ [...]
-<dd><p>Iterator function.</p>
+<dd><p>Return the next <a class="reference internal" href="#datafusion.record_batch.RecordBatch" title="datafusion.record_batch.RecordBatch"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatch</span></code></a> in the stream.</p>
</dd></dl>
<dl class="py method">
diff --git a/autoapi/datafusion/user_defined/index.html b/autoapi/datafusion/user_defined/index.html
index c2e1f6cd..df04d4b8 100644
--- a/autoapi/datafusion/user_defined/index.html
+++ b/autoapi/datafusion/user_defined/index.html
@@ -416,6 +416,11 @@
Classes
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#functions">
+ Functions
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#module-contents">
Module Contents
@@ -936,6 +941,15 @@
</li>
</ul>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.user_defined._is_pycapsule">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ _is_pycapsule()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.user_defined._R">
<code class="docutils literal notranslate">
@@ -1066,6 +1080,16 @@
</tbody>
</table>
</section>
+<section id="functions">
+<h2>Functions<a class="headerlink" href="#functions" title="Link to this heading">¶</a></h2>
+<table class="autosummary longtable table autosummary">
+<tbody>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.user_defined._is_pycapsule" title="datafusion.user_defined._is_pycapsule"><code class="xref py py-obj docutils literal notranslate"><span class="pre">_is_pycapsule</span></code></a>(→ TypeGuard[_typeshed.CapsuleType])</p></td>
+<td><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> when <code class="docutils literal notranslate"><span class="pre">value</span></code> is a CPython <code class="docutils literal notranslate"><span class="pre">PyCapsule</span></code>.</p></td>
+</tr>
+</tbody>
+</table>
+</section>
<section id="module-contents">
<h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to this heading">¶</a></h2>
<dl class="py class">
@@ -1100,7 +1124,9 @@
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.user_defined.AggregateUDF">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">AggregateUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">AggregateUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">AggregateUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em [...]
<dd><p>Class for performing scalar user-defined functions (UDF).</p>
<p>Aggregate UDFs operate on a group of rows and return a single value. See
also <a class="reference internal" href="#datafusion.user_defined.ScalarUDF" title="datafusion.user_defined.ScalarUDF"><code class="xref py py-class docutils literal notranslate"><span class="pre">ScalarUDF</span></code></a> for operating on a row by row basis.</p>
@@ -1123,7 +1149,7 @@ occur during the evaluation of the dataframe.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.AggregateUDF.from_pycapsule">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.AggregateUDFExportable [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">from_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.AggregateUDFExportable [...]
<dd><p>Create an Aggregate UDF from AggregateUDF PyCapsule object.</p>
<p>This function will instantiate a Aggregate UDF that uses a DataFusion
AggregateUDF that is exported via the FFI bindings.</p>
@@ -1131,9 +1157,13 @@ AggregateUDF that is exported via the FFI bindings.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.AggregateUDF.udaf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span> [...]
+<dt class="sig sig-object py">
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.AggregateUDFExportable" title="datafusion.user_defined.AggregateUDFExportable"><span c [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="p"> [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udaf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">accum</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">_typeshed.CapsuleType</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">& [...]
<dd><p>Create a new User-Defined Aggregate Function (UDAF).</p>
<p>This class allows you to define an aggregate function that can be used in
data aggregation or window function calls.</p>
@@ -1231,7 +1261,7 @@ aggregation or window function calls.</p>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.user_defined.ScalarUDF">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">ScalarUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em cl [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">ScalarUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em cl [...]
<dd><p>Class for performing scalar user-defined functions (UDF).</p>
<p>Scalar UDFs operate on a row by row basis. See also <a class="reference internal" href="#datafusion.user_defined.AggregateUDF" title="datafusion.user_defined.AggregateUDF"><code class="xref py py-class docutils literal notranslate"><span class="pre">AggregateUDF</span></code></a> for
operating on a group of rows.</p>
@@ -1261,9 +1291,9 @@ ScalarUDF that is exported via the FFI bindings.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.ScalarUDF.udf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">pyarrow.DataType</span><span class="p"><span c [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">pyarrow.DataType</span><span class="p"><span c [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><span class="pre"> [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><s [...]
<dt class="sig sig-object py">
<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#datafusion.user_defined.ScalarUDFExportable" title="datafusion.user_defined.ScalarUDFExportable"><span class="pr [...]
<dd><p>Create a new User-Defined Function (UDF).</p>
@@ -1333,7 +1363,7 @@ data aggregation, or window function calls.</p>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.user_defined.TableFunction">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">TableFunction</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <e [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">TableFunction</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <e [...]
<dd><p>Class for performing user-defined table functions (UDTF).</p>
<p>Table functions generate new table providers based on the
input expressions.</p>
@@ -1354,21 +1384,21 @@ descriptions.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.TableFunction._create_table_udf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><span class="p"><spa [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="pre">Ellipsis</span><spa [...]
<dd><p>Create a TableFunction instance from function arguments.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.TableFunction._create_table_udf_decorator">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span> [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_table_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span> [...]
<dd><p>Create a decorator for a WindowUDF.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.TableFunction.udtf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <spa [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <spa [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="p">< [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udtf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span>< [...]
<dd><p>Create a new User-Defined Table Function (UDTF).</p>
</dd></dl>
@@ -1606,7 +1636,7 @@ such functions can save whatever they need</p>
<dl class="py class">
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF">
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">WindowUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em cl [...]
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">WindowUDF</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em cl [...]
<dd><p>Class for performing window user-defined functions (UDF).</p>
<p>Window UDFs operate on a partition of rows. See
also <a class="reference internal" href="#datafusion.user_defined.ScalarUDF" title="datafusion.user_defined.ScalarUDF"><code class="xref py py-class docutils literal notranslate"><span class="pre">ScalarUDF</span></code></a> for operating on a row by row basis.</p>
@@ -1629,19 +1659,19 @@ occur during the evaluation of the dataframe.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF._create_window_udf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[< [...]
<dd><p>Create a WindowUDF instance from function arguments.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF._create_window_udf_decorator">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_create_window_udf_decorator</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><sp [...]
<dd><p>Create a decorator for a WindowUDF.</p>
</dd></dl>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF._get_default_name">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_get_default_name</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&# [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">_get_default_name</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig [...]
<dd><p>Get the default name for a function based on its attributes.</p>
</dd></dl>
@@ -1661,9 +1691,9 @@ WindowUDF that is exported via the FFI bindings.</p>
<dl class="py method">
<dt class="sig sig-object py" id="datafusion.user_defined.WindowUDF.udwf">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">input_types</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">pyarrow.DataType</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><sp [...]
<dt class="sig sig-object py">
-<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span><span class="p">< [...]
+<em class="property"><span class="pre">static</span><span class="w"> </span></em><span class="sig-name descname"><span class="pre">udwf</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">func</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">collections.abc.Callable</span><span class="p"><span class="pre">[</span></span><span class="p"><span class="pre">[</span></span>< [...]
<dd><p>Create a new User-Defined Window Function (UDWF).</p>
<p>This class can be used both as either a function or a decorator.</p>
<dl class="simple">
@@ -1738,6 +1768,12 @@ the online documentation for more information.</p></li>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.user_defined._is_pycapsule">
+<span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">_is_pycapsule</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">object</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-retur [...]
+<dd><p>Return <code class="docutils literal notranslate"><span class="pre">True</span></code> when <code class="docutils literal notranslate"><span class="pre">value</span></code> is a CPython <code class="docutils literal notranslate"><span class="pre">PyCapsule</span></code>.</p>
+</dd></dl>
+
<dl class="py data">
<dt class="sig sig-object py" id="datafusion.user_defined._R">
<span class="sig-prename descclassname"><span class="pre">datafusion.user_defined.</span></span><span class="sig-name descname"><span class="pre">_R</span></span><a class="headerlink" href="#datafusion.user_defined._R" title="Link to this definition">¶</a></dt>
diff --git a/genindex.html b/genindex.html
index b07a6ea9..a36dc3da 100644
--- a/genindex.html
+++ b/genindex.html
@@ -445,9 +445,11 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.__add__">(datafusion.expr.Expr method)</a>
</li>
</ul></li>
- <li><a href="autoapi/datafusion/record_batch/index.html#datafusion.record_batch.RecordBatchStream.__aiter__">__aiter__() (datafusion.record_batch.RecordBatchStream method)</a>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.__aiter__">__aiter__() (datafusion.dataframe.DataFrame method)</a>
<ul>
+ <li><a href="autoapi/datafusion/record_batch/index.html#datafusion.record_batch.RecordBatchStream.__aiter__">(datafusion.record_batch.RecordBatchStream method)</a>
+</li>
<li><a href="autoapi/datafusion/index.html#datafusion.RecordBatchStream.__aiter__">(datafusion.RecordBatchStream method)</a>
</li>
</ul></li>
@@ -464,7 +466,13 @@
</li>
</ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.ArrowArrayExportable.__arrow_c_array__">__arrow_c_array__() (datafusion.context.ArrowArrayExportable method)</a>
+
+ <ul>
+ <li><a href="autoapi/datafusion/record_batch/index.html#datafusion.record_batch.RecordBatch.__arrow_c_array__">(datafusion.record_batch.RecordBatch method)</a>
</li>
+ <li><a href="autoapi/datafusion/index.html#datafusion.RecordBatch.__arrow_c_array__">(datafusion.RecordBatch method)</a>
+</li>
+ </ul></li>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.ArrowStreamExportable.__arrow_c_stream__">__arrow_c_stream__() (datafusion.context.ArrowStreamExportable method)</a>
<ul>
@@ -533,9 +541,11 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.Expr.__invert__">(datafusion.expr.Expr method)</a>
</li>
</ul></li>
- <li><a href="autoapi/datafusion/record_batch/index.html#datafusion.record_batch.RecordBatchStream.__iter__">__iter__() (datafusion.record_batch.RecordBatchStream method)</a>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.__iter__">__iter__() (datafusion.dataframe.DataFrame method)</a>
<ul>
+ <li><a href="autoapi/datafusion/record_batch/index.html#datafusion.record_batch.RecordBatchStream.__iter__">(datafusion.record_batch.RecordBatchStream method)</a>
+</li>
<li><a href="autoapi/datafusion/index.html#datafusion.RecordBatchStream.__iter__">(datafusion.RecordBatchStream method)</a>
</li>
</ul></li>
@@ -769,6 +779,8 @@
<li><a href="autoapi/datafusion/index.html#datafusion.Table._inner">(datafusion.Table attribute)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined._is_pycapsule">_is_pycapsule() (in module datafusion.user_defined)</a>
+</li>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowUDF._normalize_input_types">_normalize_input_types() (datafusion.user_defined.WindowUDF static method)</a>
<ul>
@@ -1310,6 +1322,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.col">col() (in module datafusion.functions)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.collect">collect() (datafusion.dataframe.DataFrame method)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.collect_column">collect_column() (datafusion.dataframe.DataFrame method)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.collect_partitioned">collect_partitioned() (datafusion.dataframe.DataFrame method)</a>
</li>
@@ -2969,6 +2983,8 @@
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.select">select() (datafusion.dataframe.DataFrame method)</a>
</li>
<li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.select_columns">select_columns() (datafusion.dataframe.DataFrame method)</a>
+</li>
+ <li><a href="autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.select_exprs">select_exprs() (datafusion.dataframe.DataFrame method)</a>
</li>
<li><a href="autoapi/datafusion/substrait/index.html#datafusion.substrait.Serde">Serde (class in datafusion.substrait)</a>
</li>
diff --git a/objects.inv b/objects.inv
index 0b9573a4..a9e3c5ff 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/searchindex.js b/searchindex.js
index b8ed13a5..61e5a237 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[18, null]], "Additional Resources": [[36, "additional-resources"]], "Aggregate Functions": [[23, "aggregate-functions"], [30, "aggregate-functions"], [32, "aggregate-functions"]], "Aggregation": [[23, null]], "Alternative Approach": [[19, "alternative-approach"]], "Apache Iceberg": [[34, "apache-iceberg"]], "Arrays": [[25, "arrays"]], "Arrow": [[38, null]], "Attributes": [[4, "attributes"], [7, "attributes"], [12, "attributes"], [17, "att [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[18, null]], "Additional Resources": [[36, "additional-resources"]], "Aggregate Functions": [[23, "aggregate-functions"], [30, "aggregate-functions"], [32, "aggregate-functions"]], "Aggregation": [[23, null]], "Alternative Approach": [[19, "alternative-approach"]], "Apache Iceberg": [[34, "apache-iceberg"]], "Arrays": [[25, "arrays"]], "Arrow": [[38, null]], "Attributes": [[4, "attributes"], [7, "attributes"], [12, "attributes"], [17, "att [...]
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html b/user-guide/common-operations/aggregations.html
index fafadbdf..7c0269ad 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -610,15 +610,15 @@ Pokemon set. Since there will be many entries of <code class="docutils literal n
<span class="o">+----------+--------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+--------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span class="p">,</span> <span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span class="o">|</span> <span class="p">[]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dark</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Rock</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+----------+--------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
@@ -634,15 +634,15 @@ entries entirely removed. The second is we can use the <code class="docutils lit
<span class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Grass</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Steel</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dark</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Water</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Water</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
@@ -653,15 +653,15 @@ entries entirely removed. The second is we can use the <code class="docutils lit
<span class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Grass</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span class="o">|</span> <span class="p">[</span><span class="n">Grass</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Steel</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span class="o">|</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Dark</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
diff --git a/user-guide/common-operations/basic-info.html b/user-guide/common-operations/basic-info.html
index f88289ea..24cee81d 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -446,11 +446,11 @@
<span class="o">+-----+---------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">418</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">166</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">885</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">199</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">985</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">730</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">479</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">280</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">944</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">44</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
</pre></div>
</div>
@@ -461,8 +461,8 @@
<span class="o">+-----+--------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">418</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">166</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">730</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">479</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
</pre></div>
</div>
@@ -480,28 +480,28 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span>
<span class="n">nrs</span> <span class="n">names</span> <span class="n">random</span> <span class="n">groups</span>
-<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">418</span> <span class="n">A</span>
-<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">166</span> <span class="n">A</span>
-<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">885</span> <span class="n">B</span>
-<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">199</span> <span class="n">C</span>
-<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">985</span> <span class="n">B</span>
+<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">730</span> <span class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">479</span> <span class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">280</span> <span class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">944</span> <span class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">44</span> <span class="n">B</span>
</pre></div>
</div>
<p><a class="reference internal" href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe" title="datafusion.dataframe.DataFrame.describe"><code class="xref py py-func docutils literal notranslate"><span class="pre">describe()</span></code></a> shows a quick statistic summary of your data:</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">describe</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span>
<span class="n">DataFrame</span><span class="p">()</span>
-<span class="o">+------------+--------------------+-------+-------------------+--------+</span>
-<span class="o">|</span> <span class="n">describe</span> <span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
-<span class="o">+------------+--------------------+-------+-------------------+--------+</span>
-<span class="o">|</span> <span class="n">count</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">null_count</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">530.6</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">383.2914035039137</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">166.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">985.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">418.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">+------------+--------------------+-------+-------------------+--------+</span>
+<span class="o">+------------+--------------------+-------+------------------+--------+</span>
+<span class="o">|</span> <span class="n">describe</span> <span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
+<span class="o">+------------+--------------------+-------+------------------+--------+</span>
+<span class="o">|</span> <span class="n">count</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">null_count</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">495.4</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">355.916844220669</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">44.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">944.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">479.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">+------------+--------------------+-------+------------------+--------+</span>
</pre></div>
</div>
</section>
diff --git a/user-guide/common-operations/functions.html b/user-guide/common-operations/functions.html
index 3211e825..867876c1 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -553,20 +553,20 @@ In here we will cover some of the more popular use cases. If you want to view al
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span>
<span class="n">DataFrame</span><span class="p">()</span>
-<span class="o">+--------------------------------+</span>
-<span class="o">|</span> <span class="n">now</span><span class="p">()</span> <span class="o">|</span>
-<span class="o">+--------------------------------+</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2025</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">16</span><span class="n">T17</span><span class="p">:</span><span class="mi">13</span><span class="p">:</span><span class="mf">48.336869762</span><span class="n">Z</span> <span class="o">|</span>
-<span class="o">+--------------------------------+</span>
+<span class="o">+-------------------------------+</span>
+<span class="o">|</span> <span class="n">now</span><span class="p">()</span> <span class="o">|</span>
+<span class="o">+-------------------------------+</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">05</span><span class="n">T16</span><span class="p">:</span><span class="mi">06</span><span class="p">:</span><span class="mf">57.764994749</span> <span class="o">|</span>
+<span class="o">+-------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
</pre></div>
</div>
diff --git a/user-guide/common-operations/index.html b/user-guide/common-operations/index.html
index aa8a5bfb..5bd5dc90 100644
--- a/user-guide/common-operations/index.html
+++ b/user-guide/common-operations/index.html
@@ -446,6 +446,7 @@
<li class="toctree-l2"><a class="reference internal" href="joins.html#full-join">Full Join</a></li>
<li class="toctree-l2"><a class="reference internal" href="joins.html#left-semi-join">Left Semi Join</a></li>
<li class="toctree-l2"><a class="reference internal" href="joins.html#left-anti-join">Left Anti Join</a></li>
+<li class="toctree-l2"><a class="reference internal" href="joins.html#duplicate-keys">Duplicate Keys</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="functions.html">Functions</a><ul>
diff --git a/user-guide/common-operations/joins.html b/user-guide/common-operations/joins.html
index 56ae4eaa..20fb374e 100644
--- a/user-guide/common-operations/joins.html
+++ b/user-guide/common-operations/joins.html
@@ -432,6 +432,11 @@
Left Anti Join
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#duplicate-keys">
+ Duplicate Keys
+ </a>
+ </li>
</ul>
</nav>
@@ -571,6 +576,52 @@ the right table.</p>
</pre></div>
</div>
</section>
+<section id="duplicate-keys">
+<h2>Duplicate Keys<a class="headerlink" href="#duplicate-keys" title="Link to this heading">¶</a></h2>
+<p>It is common to join two DataFrames on a common column name. Starting in
+version 51.0.0, <code class="docutils literal notranslate"><span class="pre">datafusion-python`</span></code> will now coalesce on column with identical names by
+default. This reduces problems with ambiguous column selection after joins.
+You can disable this feature by setting the parameter <code class="docutils literal notranslate"><span class="pre">coalesce_duplicate_keys</span></code>
+to <code class="docutils literal notranslate"><span class="pre">False</span></code>.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="n">left</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pydict</span><span class="p">(</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">{</span>
+ <span class="o">....</span><span class="p">:</span> <span class="s2">"id"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span>
+ <span class="o">....</span><span class="p">:</span> <span class="s2">"customer"</span><span class="p">:</span> <span class="p">[</span><span class="s2">"Alice"</span><span class="p">,</span> <span class="s2">"Bob"</span><span class="p">,</span> <span class="s2">"Charlie"</span><span class="p">],</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">}</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">)</span>
+ <span class="o">....</span><span class="p">:</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">11</span><span class="p">]:</span> <span class="n">right</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">from_pylist</span><span class="p">([</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">{</span><span class="s2">"id"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"CityCabs"</span><span class="p">},</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">{</span><span class="s2">"id"</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span> <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"MetroRide"</span><span class="p">},</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">{</span><span class="s2">"id"</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span> <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"UrbanGo"</span><span class="p">},</span>
+ <span class="o">....</span><span class="p">:</span> <span class="p">])</span>
+ <span class="o">....</span><span class="p">:</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">12</span><span class="p">]:</span> <span class="n">left</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">right</span><span class="p">,</span> <span class="s2">"id"</span><span class="p">,</span> <span class="n">how</span><span class="o">=</span><span class="s2">"inner"</span><span class="p">)</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">12</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+----+----------+-----------+</span>
+<span class="o">|</span> <span class="nb">id</span> <span class="o">|</span> <span class="n">customer</span> <span class="o">|</span> <span class="n">name</span> <span class="o">|</span>
+<span class="o">+----+----------+-----------+</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Alice</span> <span class="o">|</span> <span class="n">CityCabs</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">Bob</span> <span class="o">|</span> <span class="n">MetroRide</span> <span class="o">|</span>
+<span class="o">+----+----------+-----------+</span>
+</pre></div>
+</div>
+<p>In contrast to the above example, if we wish to get both columns:</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">13</span><span class="p">]:</span> <span class="n">left</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">right</span><span class="p">,</span> <span class="s2">"id"</span><span class="p">,</span> <span class="n">how</span><span class="o">=</span><span class="s2">"inner"</sp [...]
+<span class="n">Out</span><span class="p">[</span><span class="mi">13</span><span class="p">]:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+----+----------+----+-----------+</span>
+<span class="o">|</span> <span class="nb">id</span> <span class="o">|</span> <span class="n">customer</span> <span class="o">|</span> <span class="nb">id</span> <span class="o">|</span> <span class="n">name</span> <span class="o">|</span>
+<span class="o">+----+----------+----+-----------+</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Alice</span> <span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">CityCabs</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">Bob</span> <span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">MetroRide</span> <span class="o">|</span>
+<span class="o">+----+----------+----+-----------+</span>
+</pre></div>
+</div>
+</section>
</section>
diff --git a/user-guide/configuration.html b/user-guide/configuration.html
index 87ebc3ee..c26e0b53 100644
--- a/user-guide/configuration.html
+++ b/user-guide/configuration.html
@@ -448,7 +448,7 @@
<div>
<section id="configuration">
-<h1>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">¶</a></h1>
+<span id="id1"></span><h1>Configuration<a class="headerlink" href="#configuration" title="Link to this heading">¶</a></h1>
<p>Let’s look at how we can configure DataFusion. When creating a <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a>, you can pass in
a <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.SessionConfig" title="datafusion.context.SessionConfig"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionConfig</span></code></a> and <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.RuntimeEnvBuilder" title="datafusion.context.RuntimeEnvBuilder"><code class="xref py py-class docutils literal notranslate"><spa [...]
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">RuntimeEnvBuilder</span><span class="p">,</span> <span class="n">SessionConfig</span><span class="p">,</span> <span class="n">SessionContext</span>
diff --git a/user-guide/dataframe/index.html b/user-guide/dataframe/index.html
index cf5c2acd..69a344d8 100644
--- a/user-guide/dataframe/index.html
+++ b/user-guide/dataframe/index.html
@@ -432,6 +432,23 @@
Terminal Operations
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#zero-copy-streaming-to-arrow-based-python-libraries">
+ Zero-copy streaming to Arrow-based Python libraries
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#pyarrow">
+ PyArrow
+ </a>
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#execute-as-stream">
+ Execute as Stream
+ </a>
+ </li>
+ </ul>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#html-rendering">
HTML Rendering
@@ -543,8 +560,9 @@ For custom data sources, see <a class="reference internal" href="../io/table_pro
<span class="c1"># Select with expressions</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">(</span><span class="n">column</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">column</span><span class="p">(</span><span class="s2">"b"</span><span class="p">),</span> <span class="n">column</span><span class="p">(</span><span class="s2">"a" [...]
-<span class="c1"># Filter rows</span>
+<span class="c1"># Filter rows (expressions or SQL strings)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">column</span><span class="p">(</span><span class="s2">"age"</span><span class="p">)</span> <span class="o">></span> <span class="n">literal</span><span class="p">(</span><span class="mi">25</span><span class="p">))</span>
+<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s2">"age > 25"</span><span class="p">)</span>
<span class="c1"># Add computed columns</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">with_column</span><span class="p">(</span><span class="s2">"full_name"</span><span class="p">,</span> <span class="n">column</span><span class="p">(</span><span class="s2">"first_name"</span><span class="p">)</span> <span class="o">+</span> <span class="n">literal</span><span class="p">(</span><span class="s2">" "</span><span class="p">)</span> [...]
@@ -634,8 +652,101 @@ as <code class="docutils literal notranslate"><span class="pre">literal()</span>
<span class="c1"># Count rows</span>
<span class="n">count</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
+
+<span class="c1"># Collect a single column of data as a PyArrow Array</span>
+<span class="n">arr</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">collect_column</span><span class="p">(</span><span class="s2">"age"</span><span class="p">)</span>
+</pre></div>
+</div>
+</section>
+<section id="zero-copy-streaming-to-arrow-based-python-libraries">
+<h2>Zero-copy streaming to Arrow-based Python libraries<a class="headerlink" href="#zero-copy-streaming-to-arrow-based-python-libraries" title="Link to this heading">¶</a></h2>
+<p>DataFusion DataFrames implement the <code class="docutils literal notranslate"><span class="pre">__arrow_c_stream__</span></code> protocol, enabling
+zero-copy, lazy streaming into Arrow-based Python libraries. With the streaming
+protocol, batches are produced on demand.</p>
+<div class="admonition note">
+<p class="admonition-title">Note</p>
+<p>The protocol is implementation-agnostic and works with any Python library
+that understands the Arrow C streaming interface (for example, PyArrow
+or other Arrow-compatible implementations). The sections below provide a
+short PyArrow-specific example and general guidance for other
+implementations.</p>
+</div>
+</section>
+<section id="pyarrow">
+<h2>PyArrow<a class="headerlink" href="#pyarrow" title="Link to this heading">¶</a></h2>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span>
+
+<span class="c1"># Create a PyArrow RecordBatchReader without materializing all batches</span>
+<span class="n">reader</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">RecordBatchReader</span><span class="o">.</span><span class="n">from_stream</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
+<span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">reader</span><span class="p">:</span>
+ <span class="o">...</span> <span class="c1"># process each batch as it is produced</span>
+</pre></div>
+</div>
+<p>DataFrames are also iterable, yielding <a class="reference internal" href="../../autoapi/datafusion/index.html#datafusion.RecordBatch" title="datafusion.RecordBatch"><code class="xref py py-class docutils literal notranslate"><span class="pre">datafusion.RecordBatch</span></code></a>
+objects lazily so you can loop over results directly without importing
+PyArrow:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">df</span><span class="p">:</span>
+ <span class="o">...</span> <span class="c1"># each batch is a ``datafusion.RecordBatch``</span>
</pre></div>
</div>
+<p>Each batch exposes <code class="docutils literal notranslate"><span class="pre">to_pyarrow()</span></code>, allowing conversion to a PyArrow
+table. <code class="docutils literal notranslate"><span class="pre">pa.table(df)</span></code> collects the entire DataFrame eagerly into a
+PyArrow table:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span>
+<span class="n">table</span> <span class="o">=</span> <span class="n">pa</span><span class="o">.</span><span class="n">table</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Asynchronous iteration is supported as well, allowing integration with
+<code class="docutils literal notranslate"><span class="pre">asyncio</span></code> event loops:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">df</span><span class="p">:</span>
+ <span class="o">...</span> <span class="c1"># process each batch as it is produced</span>
+</pre></div>
+</div>
+<p>To work with the stream directly, use <code class="docutils literal notranslate"><span class="pre">execute_stream()</span></code>, which returns a
+<a class="reference internal" href="../../autoapi/datafusion/index.html#datafusion.RecordBatchStream" title="datafusion.RecordBatchStream"><code class="xref py py-class docutils literal notranslate"><span class="pre">RecordBatchStream</span></code></a>.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">stream</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">execute_stream</span><span class="p">()</span>
+<span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span>
+ <span class="o">...</span>
+</pre></div>
+</div>
+<section id="execute-as-stream">
+<h3>Execute as Stream<a class="headerlink" href="#execute-as-stream" title="Link to this heading">¶</a></h3>
+<p>For finer control over streaming execution, use
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">execute_stream()</span></code> to obtain a
+<a class="reference internal" href="../../autoapi/datafusion/index.html#datafusion.RecordBatchStream" title="datafusion.RecordBatchStream"><code class="xref py py-class docutils literal notranslate"><span class="pre">datafusion.RecordBatchStream</span></code></a>:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">stream</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">execute_stream</span><span class="p">()</span>
+<span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span>
+ <span class="o">...</span> <span class="c1"># process each batch as it is produced</span>
+</pre></div>
+</div>
+<div class="admonition tip">
+<p class="admonition-title">Tip</p>
+<p>To get a PyArrow reader instead, call</p>
+<p><code class="docutils literal notranslate"><span class="pre">pa.RecordBatchReader.from_stream(df)</span></code>.</p>
+</div>
+<p>When partition boundaries are important,
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">execute_stream_partitioned()</span></code>
+returns an iterable of <a class="reference internal" href="../../autoapi/datafusion/index.html#datafusion.RecordBatchStream" title="datafusion.RecordBatchStream"><code class="xref py py-class docutils literal notranslate"><span class="pre">datafusion.RecordBatchStream</span></code></a> objects, one per
+partition:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">for</span> <span class="n">stream</span> <span class="ow">in</span> <span class="n">df</span><span class="o">.</span><span class="n">execute_stream_partitioned</span><span class="p">():</span>
+ <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span>
+ <span class="o">...</span> <span class="c1"># each stream yields RecordBatches</span>
+</pre></div>
+</div>
+<p>To process partitions concurrently, first collect the streams into a list
+and then poll each one in a separate <code class="docutils literal notranslate"><span class="pre">asyncio</span></code> task:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>
+
+<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">consume</span><span class="p">(</span><span class="n">stream</span><span class="p">):</span>
+ <span class="k">async</span> <span class="k">for</span> <span class="n">batch</span> <span class="ow">in</span> <span class="n">stream</span><span class="p">:</span>
+ <span class="o">...</span>
+
+<span class="n">streams</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">df</span><span class="o">.</span><span class="n">execute_stream_partitioned</span><span class="p">())</span>
+<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">consume</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">streams</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>See <a class="reference internal" href="../io/arrow.html"><span class="doc">Arrow</span></a> for additional details on the Arrow interface.</p>
+</section>
</section>
<section id="html-rendering">
<h2>HTML Rendering<a class="headerlink" href="#html-rendering" title="Link to this heading">¶</a></h2>
diff --git a/user-guide/introduction.html b/user-guide/introduction.html
index cf5fd771..9e143473 100644
--- a/user-guide/introduction.html
+++ b/user-guide/introduction.html
@@ -447,7 +447,7 @@ DataFusion through various examples and highlight the most effective ways of usi
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span><span class="w"> </span><span class="nn">datafusion</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">__version__</span>
-<span class="n">Out</span><span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="s1">'50.1.0'</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="s1">'51.0.0'</span>
</pre></div>
</div>
<p>In this documentation we will also show some examples for how DataFusion integrates
diff --git a/user-guide/io/arrow.html b/user-guide/io/arrow.html
index a4786dfe..046a0ade 100644
--- a/user-guide/io/arrow.html
+++ b/user-guide/io/arrow.html
@@ -489,16 +489,16 @@ and returns a <code class="docutils literal notranslate"><span class="pre">Struc
<h2>Exporting from DataFusion<a class="headerlink" href="#exporting-from-datafusion" title="Link to this heading">¶</a></h2>
<p>DataFusion DataFrames implement <code class="docutils literal notranslate"><span class="pre">__arrow_c_stream__</span></code> PyCapsule interface, so any
Python library that accepts these can import a DataFusion DataFrame directly.</p>
-<div class="admonition warning">
-<p class="admonition-title">Warning</p>
-<p>It is important to note that this will cause the DataFrame execution to happen, which may be
-a time consuming task. That is, you will cause a
-<a class="reference internal" href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.collect" title="datafusion.dataframe.DataFrame.collect"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.dataframe.DataFrame.collect()</span></code></a> operation call to occur.</p>
-</div>
-<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">((</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">*</span> <span class="n"> [...]
+<p>Invoking <code class="docutils literal notranslate"><span class="pre">__arrow_c_stream__</span></code> triggers execution of the underlying query, but
+batches are yielded incrementally rather than materialized all at once in memory.
+Consumers can process the stream as it arrives. The stream executes lazily,
+letting downstream readers pull batches on demand.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">col</span><span class="p">,</span> <span class="n">lit</span>
+
+<span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">select</span><span class="p">((</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">*</span> <span class="n">lit</span><span class="p">(</span><span class="mf">1.5</span><span class="p">))</spa [...]
-<span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="n">pa</span><span class="o">.</span><span class="n">table</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
-<span class="n">Out</span><span class="p">[</span><span class="mi">9</span><span class="p">]:</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="n">pa</span><span class="o">.</span><span class="n">table</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">10</span><span class="p">]:</span>
<span class="n">pyarrow</span><span class="o">.</span><span class="n">Table</span>
<span class="n">c</span><span class="p">:</span> <span class="n">double</span>
<span class="n">d</span><span class="p">:</span> <span class="n">string_view</span> <span class="ow">not</span> <span class="n">null</span>
diff --git a/user-guide/sql.html b/user-guide/sql.html
index 865fe260..6dd6e0a4 100644
--- a/user-guide/sql.html
+++ b/user-guide/sql.html
@@ -401,9 +401,19 @@
<div class="toc-item">
+<div class="tocsection onthispage pt-5 pb-3">
+ <i class="fas fa-list"></i> On this page
+</div>
<nav id="bd-toc-nav">
-
+ <ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#parameterized-queries">
+ Parameterized queries
+ </a>
+ </li>
+</ul>
+
</nav>
</div>
@@ -428,22 +438,20 @@
<p>DataFusion also offers a SQL API, read the full reference <a class="reference external" href="https://arrow.apache.org/datafusion/user-guide/sql/index.html">here</a></p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="kn">import</span><span class="w"> </span><span class="nn">datafusion</span>
-<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">col</span>
-
-<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataFrame</span><span class="p">,</span> <span class="n">SessionContext</span>
<span class="c1"># create a context</span>
-<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="n">ctx</span> <span class="o">=</span> <span class="n">datafusion</span><span class="o">.</span><span class="n">SessionContext</span><span class="p">()</span>
<span class="c1"># register a CSV</span>
-<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">register_csv</span><span class="p">(</span><span class="s1">'pokemon'</span><span class="p">,</span> <span class="s1">'pokemon.csv'</span><span class="p">)</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">register_csv</span><span class="p">(</span><span class="s2">"pokemon"</span><span class="p">,</span> <span class="s2">"pokemon.csv"</span><span class="p">)</span>
<span class="c1"># create a new statement via SQL</span>
-<span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s1">'SELECT "Attack"+"Defense", "Attack"-"Defense" FROM pokemon'</span><span class="p">)</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span><span class="s1">'SELECT "Attack"+"Defense", "Attack"-"Defense" FROM pokemon'</span><span class="p">)</span>
<span class="c1"># collect and convert to pandas DataFrame</span>
-<span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
-<span class="n">Out</span><span class="p">[</span><span class="mi">7</span><span class="p">]:</span>
+<span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
+<span class="n">Out</span><span class="p">[</span><span class="mi">6</span><span class="p">]:</span>
<span class="n">pokemon</span><span class="o">.</span><span class="n">Attack</span> <span class="o">+</span> <span class="n">pokemon</span><span class="o">.</span><span class="n">Defense</span> <span class="n">pokemon</span><span class="o">.</span><span class="n">Attack</span> <span class="o">-</span> <span class="n">pokemon</span><span class="o">.</span><span class="n">Defense</span>
<span class="mi">0</span> <span class="mi">98</span> <span class="mi">0</span>
<span class="mi">1</span> <span class="mi">125</span> <span class="o">-</span><span class="mi">1</span>
@@ -460,6 +468,114 @@
<span class="p">[</span><span class="mi">163</span> <span class="n">rows</span> <span class="n">x</span> <span class="mi">2</span> <span class="n">columns</span><span class="p">]</span>
</pre></div>
</div>
+<section id="parameterized-queries">
+<h2>Parameterized queries<a class="headerlink" href="#parameterized-queries" title="Link to this heading">¶</a></h2>
+<p>In DataFusion-Python 51.0.0 we introduced the ability to pass parameters
+in a SQL query. These are similar in concept to
+<a class="reference external" href="https://datafusion.apache.org/user-guide/sql/prepared_statements.html">prepared statements</a>,
+but allow passing named parameters into a SQL query. Consider this simple
+example.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="k">def</span><span class="w"> </span><span class="nf">show_attacks</span><span class="p">(</span><span class="n">ctx</span><span class="p">:</span> <span class="n">SessionContext</span><span class="p">,</span> <span class="n">threshold</span><span class="p">:</span> <span class="nb">int</span> [...]
+ <span class="o">...</span><span class="p">:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s1">'SELECT "Name", "Attack" FROM pokemon WHERE "Attack" > $val'</span><span class="p">,</span> <span class="n">val</span><span class="o">=</span><span class="n">threshold</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">num</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">show_attacks</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="mi">75</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+---------------------------+--------+</span>
+<span class="o">|</span> <span class="n">Name</span> <span class="o">|</span> <span class="n">Attack</span> <span class="o">|</span>
+<span class="o">+---------------------------+--------+</span>
+<span class="o">|</span> <span class="n">Venusaur</span> <span class="o">|</span> <span class="mi">82</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">VenusaurMega</span> <span class="n">Venusaur</span> <span class="o">|</span> <span class="mi">100</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Charizard</span> <span class="o">|</span> <span class="mi">84</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">X</span> <span class="o">|</span> <span class="mi">130</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">Y</span> <span class="o">|</span> <span class="mi">104</span> <span class="o">|</span>
+<span class="o">+---------------------------+--------+</span>
+</pre></div>
+</div>
+<p>When passing parameters like the example above we convert the Python objects
+into their string representation. We also have special case handling
+for <a class="reference internal" href="../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a> objects, since they cannot simply
+be turned into string representations for an SQL query. In these cases we
+will register a temporary view in the <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a>
+using a generated table name.</p>
+<p>The formatting for passing string replacement objects is to precede the
+variable name with a single <code class="docutils literal notranslate"><span class="pre">$</span></code>. This works for all dialects in
+the SQL parser except <code class="docutils literal notranslate"><span class="pre">hive</span></code> and <code class="docutils literal notranslate"><span class="pre">mysql</span></code>. Since these dialects do not
+support named placeholders, we are unable to do this type of replacement.
+We recommend either switching to another dialect or using Python
+f-string style replacement.</p>
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>To support DataFrame parameterized queries, your session must support
+registration of temporary views. The default
+<a class="reference internal" href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.CatalogProvider" title="datafusion.catalog.CatalogProvider"><code class="xref py py-class docutils literal notranslate"><span class="pre">CatalogProvider</span></code></a> and
+<a class="reference internal" href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.SchemaProvider" title="datafusion.catalog.SchemaProvider"><code class="xref py py-class docutils literal notranslate"><span class="pre">SchemaProvider</span></code></a> do have this capability.
+If you have implemented custom providers, it is important that temporary
+views do not persist across <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a>
+or you may get unintended consequences.</p>
+</div>
+<p>The following example shows passing in both a <a class="reference internal" href="../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame" title="datafusion.dataframe.DataFrame"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFrame</span></code></a>
+object as well as a Python object to be used in parameterized replacement.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="k">def</span><span class="w"> </span><span class="nf">show_column</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">ctx</span><span class="p">:</span> <span class="n">SessionContext</span><span class="p">,</span> <span class="n">column</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">df</span><span class="p">:</span> <span class="n">DataFrame</span><span class="p">,</span> <span class="n">threshold</span><span class="p">:</span> <span class="nb">int</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s1">'SELECT "Name", $col FROM $df WHERE $col > $val'</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">col</span><span class="o">=</span><span class="n">column</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">df</span><span class="o">=</span><span class="n">df</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">val</span><span class="o">=</span><span class="n">threshold</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">num</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">df</span> <span class="o">=</span> <span class="n">ctx</span><span class="o">.</span><span class="n">table</span><span class="p">(</span><span class="s2">"pokemon"</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">show_column</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="s1">'"Defense"'</span><span class="p">,</span> <span class="n">df</span><span class="p">,</span> <span class="mi">75</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+---------------------------+---------+</span>
+<span class="o">|</span> <span class="n">Name</span> <span class="o">|</span> <span class="n">Defense</span> <span class="o">|</span>
+<span class="o">+---------------------------+---------+</span>
+<span class="o">|</span> <span class="n">Venusaur</span> <span class="o">|</span> <span class="mi">83</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">VenusaurMega</span> <span class="n">Venusaur</span> <span class="o">|</span> <span class="mi">123</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Charizard</span> <span class="o">|</span> <span class="mi">78</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">X</span> <span class="o">|</span> <span class="mi">111</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">Y</span> <span class="o">|</span> <span class="mi">78</span> <span class="o">|</span>
+<span class="o">+---------------------------+---------+</span>
+</pre></div>
+</div>
+<p>The approach implemented for conversion of variables into a SQL query
+relies on string conversion. This has the potential for data loss,
+specifically for cases like floating point numbers. If you need to pass
+variables into a parameterized query and it is important to maintain the
+original value without conversion to a string, then you can use the
+optional parameter <code class="docutils literal notranslate"><span class="pre">param_values</span></code> to specify these. This parameter
+expects a dictionary mapping from the parameter name to a Python
+object. Those objects will be cast into a
+<a class="reference external" href="https://arrow.apache.org/docs/python/generated/pyarrow.Scalar.html">PyArrow Scalar Value</a>.</p>
+<p>Using <code class="docutils literal notranslate"><span class="pre">param_values</span></code> will rely on the SQL dialect you have configured
+for your session. This can be set using the <a class="reference internal" href="configuration.html#configuration"><span class="std std-ref">configuration options</span></a>
+of your <a class="reference internal" href="../autoapi/datafusion/context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a>. Similar to how
+<a class="reference external" href="https://datafusion.apache.org/user-guide/sql/prepared_statements.html">prepared statements</a>
+work, these parameters are limited to places where you would pass in a
+scalar value, such as a comparison.</p>
+<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="k">def</span><span class="w"> </span><span class="nf">param_attacks</span><span class="p">(</span><span class="n">ctx</span><span class="p">:</span> <span class="n">SessionContext</span><span class="p">,</span> <span class="n">threshold</span><span class="p">:</span> <span class="nb">int</span [...]
+ <span class="o">...</span><span class="p">:</span> <span class="n">ctx</span><span class="o">.</span><span class="n">sql</span><span class="p">(</span>
+ <span class="o">...</span><span class="p">:</span> <span class="s1">'SELECT "Name", "Attack" FROM pokemon WHERE "Attack" > $val'</span><span class="p">,</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">param_values</span><span class="o">=</span><span class="p">{</span><span class="s2">"val"</span><span class="p">:</span> <span class="n">threshold</span><span class="p">},</span>
+ <span class="o">...</span><span class="p">:</span> <span class="p">)</span><span class="o">.</span><span class="n">show</span><span class="p">(</span><span class="n">num</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span> <span class="n">param_attacks</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span> <span class="mi">75</span><span class="p">)</span>
+ <span class="o">...</span><span class="p">:</span>
+<span class="n">DataFrame</span><span class="p">()</span>
+<span class="o">+---------------------------+--------+</span>
+<span class="o">|</span> <span class="n">Name</span> <span class="o">|</span> <span class="n">Attack</span> <span class="o">|</span>
+<span class="o">+---------------------------+--------+</span>
+<span class="o">|</span> <span class="n">Venusaur</span> <span class="o">|</span> <span class="mi">82</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">VenusaurMega</span> <span class="n">Venusaur</span> <span class="o">|</span> <span class="mi">100</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Charizard</span> <span class="o">|</span> <span class="mi">84</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">X</span> <span class="o">|</span> <span class="mi">130</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">CharizardMega</span> <span class="n">Charizard</span> <span class="n">Y</span> <span class="o">|</span> <span class="mi">104</span> <span class="o">|</span>
+<span class="o">+---------------------------+--------+</span>
+</pre></div>
+</div>
+</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org