You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/05/26 20:01:46 UTC
(datafusion-python) branch asf-staging updated: Publish built docs triggered by 081325afe3ac97c6d2ef793a352a26a2634d2738
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 2e78b405 Publish built docs triggered by 081325afe3ac97c6d2ef793a352a26a2634d2738
2e78b405 is described below
commit 2e78b4059942d656b144a318461aea1c8e6b25ae
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Tue May 26 20:01:42 2026 +0000
Publish built docs triggered by 081325afe3ac97c6d2ef793a352a26a2634d2738
---
_sources/autoapi/datafusion/context/index.rst.txt | 57 ++++++++++
_sources/autoapi/datafusion/expr/index.rst.txt | 30 +++--
_sources/autoapi/datafusion/index.rst.txt | 30 +++--
_sources/autoapi/datafusion/ipc/index.rst.txt | 92 +++++++++++++++-
autoapi/datafusion/context/index.html | 64 +++++++++++
autoapi/datafusion/expr/index.html | 27 +++--
autoapi/datafusion/index.html | 27 +++--
autoapi/datafusion/ipc/index.html | 127 +++++++++++++++++++++-
genindex.html | 12 +-
objects.inv | Bin 10799 -> 10823 bytes
searchindex.js | 2 +-
user-guide/common-operations/aggregations.html | 34 +++---
user-guide/common-operations/basic-info.html | 34 +++---
user-guide/common-operations/functions.html | 20 ++--
user-guide/common-operations/udf-and-udfa.html | 4 +-
15 files changed, 477 insertions(+), 83 deletions(-)
diff --git a/_sources/autoapi/datafusion/context/index.rst.txt b/_sources/autoapi/datafusion/context/index.rst.txt
index e4d9c42b..8e217339 100644
--- a/_sources/autoapi/datafusion/context/index.rst.txt
+++ b/_sources/autoapi/datafusion/context/index.rst.txt
@@ -1483,6 +1483,63 @@ Module Contents
+ .. py:method:: with_python_udf_inlining(*, enabled: bool) -> SessionContext
+
+ Control whether Python UDFs are embedded in serialized expressions.
+
+ ``enabled`` is keyword-only and required: callers must pick a
+ mode explicitly. Fresh sessions inline UDFs (``enabled=True``
+ behavior) until this method overrides the toggle.
+
+ With ``enabled=True``, serialized expressions carry the Python
+ code for any scalar, aggregate, or window UDFs they reference.
+ The receiver rebuilds the UDFs from those bytes and does not
+ need to register them first.
+
+ With ``enabled=False``, serialized expressions store only the
+ UDF names. This has two uses:
+
+ * **Cross-language portability.** The bytes can be decoded by a
+ non-Python receiver, which must already have UDFs registered
+ under matching names.
+ * **Safer deserialization.** :meth:`Expr.from_bytes` will refuse
+ to rebuild Python UDFs rather than call ``cloudpickle.loads``
+ on untrusted input.
+
+ The setting affects :meth:`Expr.to_bytes` and
+ :meth:`Expr.from_bytes` whenever this session is passed as the
+ ``ctx`` argument. :func:`pickle.dumps` and :func:`pickle.loads`
+ do not pass a context, so to apply the setting through pickle,
+ register this session with
+ :func:`datafusion.ipc.set_sender_ctx` on the sender and
+ :func:`datafusion.ipc.set_worker_ctx` on the receiver.
+
+ .. warning:: Security
+ This setting narrows only :meth:`Expr.from_bytes`. Calling
+ :func:`pickle.loads` on untrusted bytes remains unsafe
+ regardless of the toggle.
+
+ Returns a new :class:`SessionContext` with the toggle applied;
+ the original session is unchanged.
+
+ .. rubric:: Examples
+
+ >>> import pyarrow as pa
+ >>> from datafusion import SessionContext, Expr, col, udf
+ >>> ctx = SessionContext()
+ >>> identity = udf(lambda a: a, [pa.int64()], pa.int64(),
+ ... volatility="immutable", name="identity_demo")
+ >>> ctx.register_udf(identity)
+ >>> blob = identity(col("x")).to_bytes(ctx)
+ >>> strict = SessionContext().with_python_udf_inlining(enabled=False)
+ >>> try:
+ ... Expr.from_bytes(blob, strict)
+ ... except Exception as e:
+ ... print("Refusing to deserialize" in str(e))
+ True
+
+
+
.. py:attribute:: ctx
diff --git a/_sources/autoapi/datafusion/expr/index.rst.txt b/_sources/autoapi/datafusion/expr/index.rst.txt
index 21702da8..154a441b 100644
--- a/_sources/autoapi/datafusion/expr/index.rst.txt
+++ b/_sources/autoapi/datafusion/expr/index.rst.txt
@@ -348,6 +348,13 @@ Module Contents
>>> pickle.loads(pickle.dumps(e)).canonical_name()
'a * Int64(2)'
+ The encoding side honors a driver-side sender context installed
+ via :func:`datafusion.ipc.set_sender_ctx` — that is how
+ :meth:`SessionContext.with_python_udf_inlining` propagates
+ through ``pickle.dumps``. The sender context is read by
+ ``__reduce__``, so :func:`copy.copy` and :func:`copy.deepcopy`
+ — which also go through ``__reduce__`` — pick it up too.
+
.. py:method:: __repr__() -> str
@@ -688,7 +695,9 @@ Module Contents
Accepts output of :meth:`to_bytes` or :func:`pickle.dumps`.
``ctx`` is the :class:`SessionContext` used to resolve any
- function references that travel by name (e.g. FFI UDFs). When
+ function references that travel by name (e.g. FFI UDFs, or
+ Python UDFs sent with inlining disabled via
+ :meth:`SessionContext.with_python_udf_inlining`). When
``ctx`` is ``None`` the worker context installed via
:func:`datafusion.ipc.set_worker_ctx` is consulted; if no worker
context is installed, the global :class:`SessionContext` is used
@@ -1056,13 +1065,18 @@ Module Contents
worker process for distributed evaluation.
When ``ctx`` is supplied, encoding routes through that session's
- installed :class:`LogicalExtensionCodec`. When ``ctx`` is
- ``None``, the default codec is used.
-
- Built-in functions and Python UDFs (scalar, aggregate, window)
- travel inside the returned bytes; the worker does not need to
- pre-register them. UDFs imported via the FFI capsule protocol
- travel by name only and must be registered on the worker.
+ installed :class:`LogicalExtensionCodec` (so settings like
+ :meth:`SessionContext.with_python_udf_inlining` take effect).
+ When ``ctx`` is ``None``, the default codec is used (Python UDF
+ inlining on, no user-installed extension codec).
+
+ Built-in functions travel inside the returned bytes. Python UDFs
+ (scalar, aggregate, window) also inline by default, so the worker
+ does not need to pre-register them; when the encoding session has
+ :meth:`SessionContext.with_python_udf_inlining` set to ``False``,
+ Python UDFs travel by name only and must be registered on the
+ worker. UDFs imported via the FFI capsule protocol always travel
+ by name only and must be registered on the worker.
.. warning:: Security
Bytes returned here may embed a cloudpickled Python
diff --git a/_sources/autoapi/datafusion/index.rst.txt b/_sources/autoapi/datafusion/index.rst.txt
index 2a07294e..cb84080d 100644
--- a/_sources/autoapi/datafusion/index.rst.txt
+++ b/_sources/autoapi/datafusion/index.rst.txt
@@ -946,6 +946,13 @@ Package Contents
>>> pickle.loads(pickle.dumps(e)).canonical_name()
'a * Int64(2)'
+ The encoding side honors a driver-side sender context installed
+ via :func:`datafusion.ipc.set_sender_ctx` — that is how
+ :meth:`SessionContext.with_python_udf_inlining` propagates
+ through ``pickle.dumps``. The sender context is read by
+ ``__reduce__``, so :func:`copy.copy` and :func:`copy.deepcopy`
+ — which also go through ``__reduce__`` — pick it up too.
+
.. py:method:: __repr__() -> str
@@ -1286,7 +1293,9 @@ Package Contents
Accepts output of :meth:`to_bytes` or :func:`pickle.dumps`.
``ctx`` is the :class:`SessionContext` used to resolve any
- function references that travel by name (e.g. FFI UDFs). When
+ function references that travel by name (e.g. FFI UDFs, or
+ Python UDFs sent with inlining disabled via
+ :meth:`SessionContext.with_python_udf_inlining`). When
``ctx`` is ``None`` the worker context installed via
:func:`datafusion.ipc.set_worker_ctx` is consulted; if no worker
context is installed, the global :class:`SessionContext` is used
@@ -1654,13 +1663,18 @@ Package Contents
worker process for distributed evaluation.
When ``ctx`` is supplied, encoding routes through that session's
- installed :class:`LogicalExtensionCodec`. When ``ctx`` is
- ``None``, the default codec is used.
-
- Built-in functions and Python UDFs (scalar, aggregate, window)
- travel inside the returned bytes; the worker does not need to
- pre-register them. UDFs imported via the FFI capsule protocol
- travel by name only and must be registered on the worker.
+ installed :class:`LogicalExtensionCodec` (so settings like
+ :meth:`SessionContext.with_python_udf_inlining` take effect).
+ When ``ctx`` is ``None``, the default codec is used (Python UDF
+ inlining on, no user-installed extension codec).
+
+ Built-in functions travel inside the returned bytes. Python UDFs
+ (scalar, aggregate, window) also inline by default, so the worker
+ does not need to pre-register them; when the encoding session has
+ :meth:`SessionContext.with_python_udf_inlining` set to ``False``,
+ Python UDFs travel by name only and must be registered on the
+ worker. UDFs imported via the FFI capsule protocol always travel
+ by name only and must be registered on the worker.
.. warning:: Security
Bytes returned here may embed a cloudpickled Python
diff --git a/_sources/autoapi/datafusion/ipc/index.rst.txt b/_sources/autoapi/datafusion/ipc/index.rst.txt
index 26fcc9eb..d81b3443 100644
--- a/_sources/autoapi/datafusion/ipc/index.rst.txt
+++ b/_sources/autoapi/datafusion/ipc/index.rst.txt
@@ -5,7 +5,7 @@ datafusion.ipc
.. autoapi-nested-parse::
- Worker-side setup for distributing DataFusion expressions.
+ Driver- and worker-side setup for distributing DataFusion expressions.
When a :class:`Expr` is shipped to a worker process (e.g. through
:func:`multiprocessing.Pool` or a Ray actor), the worker reconstructs the
@@ -44,6 +44,36 @@ datafusion.ipc
:meth:`datafusion.Expr.to_bytes` for examples of what travels by
value vs. by reference.
+ On the driver side, call :func:`set_sender_ctx` to control how
+ :func:`pickle.dumps` encodes expressions — for example, to apply
+ :meth:`SessionContext.with_python_udf_inlining` to every pickled
+ expression on this thread:
+
+ >>> import pickle
+ >>> from datafusion import SessionContext, col, lit
+ >>> from datafusion.ipc import clear_sender_ctx, set_sender_ctx
+ >>> driver_ctx = SessionContext().with_python_udf_inlining(enabled=False)
+ >>> set_sender_ctx(driver_ctx)
+ >>> try:
+ ... blob = pickle.dumps(col("a") + lit(1))
+ ... finally:
+ ... clear_sender_ctx()
+ >>> isinstance(blob, bytes)
+ True
+
+ Without a sender context the default codec is used (Python UDF
+ inlining on). The sender context only affects pickle / ``to_bytes``
+ encoding; explicit ``expr.to_bytes(ctx)`` calls still use the supplied
+ ``ctx``.
+
+ The thread-local sender context holds a strong reference to the
+ installed :class:`SessionContext` until :func:`clear_sender_ctx` is
+ called or the thread exits. Long-running driver threads that install a sender
+ context once and never clear it will retain that session for the
+ lifetime of the thread; pair :func:`set_sender_ctx` with
+ :func:`clear_sender_ctx` (e.g. in a ``try``/``finally``) when the
+ sender context is only needed for a bounded scope.
+
Functions
@@ -51,14 +81,36 @@ Functions
.. autoapisummary::
+ datafusion.ipc.clear_sender_ctx
datafusion.ipc.clear_worker_ctx
+ datafusion.ipc.get_sender_ctx
datafusion.ipc.get_worker_ctx
+ datafusion.ipc.set_sender_ctx
datafusion.ipc.set_worker_ctx
Module Contents
---------------
+.. py:function:: clear_sender_ctx() -> None
+
+ Remove this driver's installed sender :class:`SessionContext`.
+
+ After clearing, pickled expressions fall back to the default codec
+ (Python UDF inlining on).
+
+ .. rubric:: Examples
+
+ >>> from datafusion import SessionContext
+ >>> from datafusion.ipc import (
+ ... set_sender_ctx, clear_sender_ctx, get_sender_ctx,
+ ... )
+ >>> set_sender_ctx(SessionContext())
+ >>> clear_sender_ctx()
+ >>> get_sender_ctx() is None
+ True
+
+
.. py:function:: clear_worker_ctx() -> None
Remove this worker's installed :class:`SessionContext`.
@@ -78,6 +130,18 @@ Module Contents
True
+.. py:function:: get_sender_ctx() -> datafusion.context.SessionContext | None
+
+ Return this driver's installed sender :class:`SessionContext`, or ``None``.
+
+ .. rubric:: Examples
+
+ >>> from datafusion.ipc import get_sender_ctx, clear_sender_ctx
+ >>> clear_sender_ctx()
+ >>> get_sender_ctx() is None
+ True
+
+
.. py:function:: get_worker_ctx() -> datafusion.context.SessionContext | None
Return this worker's installed :class:`SessionContext`, or ``None``.
@@ -90,6 +154,32 @@ Module Contents
True
+.. py:function:: set_sender_ctx(ctx: datafusion.context.SessionContext) -> None
+
+ Install this driver's :class:`SessionContext` for outbound pickles.
+
+ Controls how :func:`pickle.dumps` encodes :class:`Expr` instances on
+ this thread. The most useful application is propagating a session
+ configured with
+ :meth:`SessionContext.with_python_udf_inlining` so the toggle takes
+ effect through pickle (which otherwise calls
+ :meth:`Expr.to_bytes` with no context and uses the default codec).
+
+ Idempotent: overwrites any previous value. Stored in a thread-local
+ slot, so worker threads on the driver may install their own contexts.
+ Does not affect :meth:`Expr.to_bytes` calls that pass an explicit
+ ``ctx`` — those continue to use the supplied context.
+
+ .. rubric:: Examples
+
+ >>> from datafusion import SessionContext
+ >>> from datafusion.ipc import set_sender_ctx, get_sender_ctx
+ >>> driver = SessionContext().with_python_udf_inlining(enabled=False)
+ >>> set_sender_ctx(driver)
+ >>> get_sender_ctx() is driver
+ True
+
+
.. py:function:: set_worker_ctx(ctx: datafusion.context.SessionContext) -> None
Install this worker's :class:`SessionContext` for shipped expressions.
diff --git a/autoapi/datafusion/context/index.html b/autoapi/datafusion/context/index.html
index e51f606c..66d2de87 100644
--- a/autoapi/datafusion/context/index.html
+++ b/autoapi/datafusion/context/index.html
@@ -1428,6 +1428,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.context.SessionContext.with_python_udf_inlining">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionContext.with_python_udf_inlining()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.context.SessionContext.ctx">
<code class="docutils literal notranslate">
@@ -3242,6 +3251,61 @@ functions.</p>
<a class="reference internal" href="../user_defined/index.html#datafusion.user_defined.PhysicalExtensionCodecExportable" title="datafusion.user_defined.PhysicalExtensionCodecExportable"><code class="xref py py-class docutils literal notranslate"><span class="pre">PhysicalExtensionCodecExportable</span></code></a>).</p>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.context.SessionContext.with_python_udf_inlining">
+<span class="sig-name descname"><span class="pre">with_python_udf_inlining</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">enabled</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">bool</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon" [...]
+<dd><p>Control whether Python UDFs are embedded in serialized expressions.</p>
+<p><code class="docutils literal notranslate"><span class="pre">enabled</span></code> is keyword-only and required: callers must pick a
+mode explicitly. Fresh sessions inline UDFs (<code class="docutils literal notranslate"><span class="pre">enabled=True</span></code>
+behavior) until this method overrides the toggle.</p>
+<p>With <code class="docutils literal notranslate"><span class="pre">enabled=True</span></code>, serialized expressions carry the Python
+code for any scalar, aggregate, or window UDFs they reference.
+The receiver rebuilds the UDFs from those bytes and does not
+need to register them first.</p>
+<p>With <code class="docutils literal notranslate"><span class="pre">enabled=False</span></code>, serialized expressions store only the
+UDF names. This has two uses:</p>
+<ul class="simple">
+<li><p><strong>Cross-language portability.</strong> The bytes can be decoded by a
+non-Python receiver, which must already have UDFs registered
+under matching names.</p></li>
+<li><p><strong>Safer deserialization.</strong> <code class="xref py py-meth docutils literal notranslate"><span class="pre">Expr.from_bytes()</span></code> will refuse
+to rebuild Python UDFs rather than call <code class="docutils literal notranslate"><span class="pre">cloudpickle.loads</span></code>
+on untrusted input.</p></li>
+</ul>
+<p>The setting affects <code class="xref py py-meth docutils literal notranslate"><span class="pre">Expr.to_bytes()</span></code> and
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">Expr.from_bytes()</span></code> whenever this session is passed as the
+<code class="docutils literal notranslate"><span class="pre">ctx</span></code> argument. <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code> and <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.loads()</span></code>
+do not pass a context, so to apply the setting through pickle,
+register this session with
+<a class="reference internal" href="../ipc/index.html#datafusion.ipc.set_sender_ctx" title="datafusion.ipc.set_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.ipc.set_sender_ctx()</span></code></a> on the sender and
+<a class="reference internal" href="../ipc/index.html#datafusion.ipc.set_worker_ctx" title="datafusion.ipc.set_worker_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.ipc.set_worker_ctx()</span></code></a> on the receiver.</p>
+<div class="admonition warning">
+<p class="admonition-title">Warning</p>
+<p>Security
+This setting narrows only <code class="xref py py-meth docutils literal notranslate"><span class="pre">Expr.from_bytes()</span></code>. Calling
+<code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.loads()</span></code> on untrusted bytes remains unsafe
+regardless of the toggle.</p>
+</div>
+<p>Returns a new <a class="reference internal" href="#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code></a> with the toggle applied;
+the original session is unchanged.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">pa</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">SessionContext</span><span class="p">,</span> <span class="n">Expr</span><span class="p">,</span> <span class="n">col</span><span class="p">,</span> <span class="n">udf</span>
+<span class="gp">>>> </span><span class="n">ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">identity</span> <span class="o">=</span> <span class="n">udf</span><span class="p">(</span><span class="k">lambda</span> <span class="n">a</span><span class="p">:</span> <span class="n">a</span><span class="p">,</span> <span class="p">[</span><span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p">()],</span> <span class="n">pa</span><span class="o">.</span><span class="n">int64</span><span class="p [...]
+<span class="gp">... </span> <span class="n">volatility</span><span class="o">=</span><span class="s2">"immutable"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">"identity_demo"</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">ctx</span><span class="o">.</span><span class="n">register_udf</span><span class="p">(</span><span class="n">identity</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">blob</span> <span class="o">=</span> <span class="n">identity</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"x"</span><span class="p">))</span><span class="o">.</span><span class="n">to_bytes</span><span class="p">(</span><span class="n">ctx</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">strict</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span><span class="o">.</span><span class="n">with_python_udf_inlining</span><span class="p">(</span><span class="n">enabled</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="k">try</span><span class="p">:</span>
+<span class="gp">... </span> <span class="n">Expr</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">blob</span><span class="p">,</span> <span class="n">strict</span><span class="p">)</span>
+<span class="gp">... </span><span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+<span class="gp">... </span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Refusing to deserialize"</span> <span class="ow">in</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+<span class="go">True</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py attribute">
<dt class="sig sig-object py" id="datafusion.context.SessionContext.ctx">
<span class="sig-name descname"><span class="pre">ctx</span></span><a class="headerlink" href="#datafusion.context.SessionContext.ctx" title="Link to this definition">¶</a></dt>
diff --git a/autoapi/datafusion/expr/index.html b/autoapi/datafusion/expr/index.html
index e0221349..42b09d02 100644
--- a/autoapi/datafusion/expr/index.html
+++ b/autoapi/datafusion/expr/index.html
@@ -3205,6 +3205,12 @@ details on what travels by value vs. by reference.</p>
<span class="go">'a * Int64(2)'</span>
</pre></div>
</div>
+<p>The encoding side honors a driver-side sender context installed
+via <a class="reference internal" href="../ipc/index.html#datafusion.ipc.set_sender_ctx" title="datafusion.ipc.set_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.ipc.set_sender_ctx()</span></code></a> — that is how
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> propagates
+through <code class="docutils literal notranslate"><span class="pre">pickle.dumps</span></code>. The sender context is read by
+<code class="docutils literal notranslate"><span class="pre">__reduce__</span></code>, so <code class="xref py py-func docutils literal notranslate"><span class="pre">copy.copy()</span></code> and <code class="xref py py-func docutils literal notranslate"><span class="pre">copy.deepcopy()</span></code>
+— which also go through <code class="docutils literal notranslate"><span class="pre">__reduce__</span></code> — pick it up too.</p>
</dd></dl>
<dl class="py method">
@@ -3550,7 +3556,9 @@ type of expression, an error will be generated when <code class="docutils litera
<dd><p>Reconstruct an expression from serialized bytes.</p>
<p>Accepts output of <a class="reference internal" href="#datafusion.expr.Expr.to_bytes" title="datafusion.expr.Expr.to_bytes"><code class="xref py py-meth docutils literal notranslate"><span class="pre">to_bytes()</span></code></a> or <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code>.
<code class="docutils literal notranslate"><span class="pre">ctx</span></code> is the <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> used to resolve any
-function references that travel by name (e.g. FFI UDFs). When
+function references that travel by name (e.g. FFI UDFs, or
+Python UDFs sent with inlining disabled via
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code>). When
<code class="docutils literal notranslate"><span class="pre">ctx</span></code> is <code class="docutils literal notranslate"><span class="pre">None</span></code> the worker context installed via
<a class="reference internal" href="../ipc/index.html#datafusion.ipc.set_worker_ctx" title="datafusion.ipc.set_worker_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.ipc.set_worker_ctx()</span></code></a> is consulted; if no worker
context is installed, the global <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> is used
@@ -3914,12 +3922,17 @@ UTF8View literal, like in:
<p>Use this — or <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code> — to send an expression to a
worker process for distributed evaluation.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">ctx</span></code> is supplied, encoding routes through that session’s
-installed <code class="xref py py-class docutils literal notranslate"><span class="pre">LogicalExtensionCodec</span></code>. When <code class="docutils literal notranslate"><span class="pre">ctx</span></code> is
-<code class="docutils literal notranslate"><span class="pre">None</span></code>, the default codec is used.</p>
-<p>Built-in functions and Python UDFs (scalar, aggregate, window)
-travel inside the returned bytes; the worker does not need to
-pre-register them. UDFs imported via the FFI capsule protocol
-travel by name only and must be registered on the worker.</p>
+installed <code class="xref py py-class docutils literal notranslate"><span class="pre">LogicalExtensionCodec</span></code> (so settings like
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> take effect).
+When <code class="docutils literal notranslate"><span class="pre">ctx</span></code> is <code class="docutils literal notranslate"><span class="pre">None</span></code>, the default codec is used (Python UDF
+inlining on, no user-installed extension codec).</p>
+<p>Built-in functions travel inside the returned bytes. Python UDFs
+(scalar, aggregate, window) also inline by default, so the worker
+does not need to pre-register them; when the encoding session has
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> set to <code class="docutils literal notranslate"><span class="pre">False</span></code>,
+Python UDFs travel by name only and must be registered on the
+worker. UDFs imported via the FFI capsule protocol always travel
+by name only and must be registered on the worker.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Security
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index 6d926575..75fdc469 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -4795,6 +4795,12 @@ details on what travels by value vs. by reference.</p>
<span class="go">'a * Int64(2)'</span>
</pre></div>
</div>
+<p>The encoding side honors a driver-side sender context installed
+via <a class="reference internal" href="ipc/index.html#datafusion.ipc.set_sender_ctx" title="datafusion.ipc.set_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.ipc.set_sender_ctx()</span></code></a> — that is how
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> propagates
+through <code class="docutils literal notranslate"><span class="pre">pickle.dumps</span></code>. The sender context is read by
+<code class="docutils literal notranslate"><span class="pre">__reduce__</span></code>, so <code class="xref py py-func docutils literal notranslate"><span class="pre">copy.copy()</span></code> and <code class="xref py py-func docutils literal notranslate"><span class="pre">copy.deepcopy()</span></code>
+— which also go through <code class="docutils literal notranslate"><span class="pre">__reduce__</span></code> — pick it up too.</p>
</dd></dl>
<dl class="py method">
@@ -5140,7 +5146,9 @@ type of expression, an error will be generated when <code class="docutils litera
<dd><p>Reconstruct an expression from serialized bytes.</p>
<p>Accepts output of <a class="reference internal" href="#datafusion.Expr.to_bytes" title="datafusion.Expr.to_bytes"><code class="xref py py-meth docutils literal notranslate"><span class="pre">to_bytes()</span></code></a> or <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code>.
<code class="docutils literal notranslate"><span class="pre">ctx</span></code> is the <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> used to resolve any
-function references that travel by name (e.g. FFI UDFs). When
+function references that travel by name (e.g. FFI UDFs, or
+Python UDFs sent with inlining disabled via
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code>). When
<code class="docutils literal notranslate"><span class="pre">ctx</span></code> is <code class="docutils literal notranslate"><span class="pre">None</span></code> the worker context installed via
<a class="reference internal" href="ipc/index.html#datafusion.ipc.set_worker_ctx" title="datafusion.ipc.set_worker_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">datafusion.ipc.set_worker_ctx()</span></code></a> is consulted; if no worker
context is installed, the global <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> is used
@@ -5504,12 +5512,17 @@ UTF8View literal, like in:
<p>Use this — or <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code> — to send an expression to a
worker process for distributed evaluation.</p>
<p>When <code class="docutils literal notranslate"><span class="pre">ctx</span></code> is supplied, encoding routes through that session’s
-installed <code class="xref py py-class docutils literal notranslate"><span class="pre">LogicalExtensionCodec</span></code>. When <code class="docutils literal notranslate"><span class="pre">ctx</span></code> is
-<code class="docutils literal notranslate"><span class="pre">None</span></code>, the default codec is used.</p>
-<p>Built-in functions and Python UDFs (scalar, aggregate, window)
-travel inside the returned bytes; the worker does not need to
-pre-register them. UDFs imported via the FFI capsule protocol
-travel by name only and must be registered on the worker.</p>
+installed <code class="xref py py-class docutils literal notranslate"><span class="pre">LogicalExtensionCodec</span></code> (so settings like
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> take effect).
+When <code class="docutils literal notranslate"><span class="pre">ctx</span></code> is <code class="docutils literal notranslate"><span class="pre">None</span></code>, the default codec is used (Python UDF
+inlining on, no user-installed extension codec).</p>
+<p>Built-in functions travel inside the returned bytes. Python UDFs
+(scalar, aggregate, window) also inline by default, so the worker
+does not need to pre-register them; when the encoding session has
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> set to <code class="docutils literal notranslate"><span class="pre">False</span></code>,
+Python UDFs travel by name only and must be registered on the
+worker. UDFs imported via the FFI capsule protocol always travel
+by name only and must be registered on the worker.</p>
<div class="admonition warning">
<p class="admonition-title">Warning</p>
<p>Security
diff --git a/autoapi/datafusion/ipc/index.html b/autoapi/datafusion/ipc/index.html
index 9b0d23b4..5202a92c 100644
--- a/autoapi/datafusion/ipc/index.html
+++ b/autoapi/datafusion/ipc/index.html
@@ -437,6 +437,15 @@
Module Contents
</a>
<ul class="visible nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.ipc.clear_sender_ctx">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ clear_sender_ctx()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.ipc.clear_worker_ctx">
<code class="docutils literal notranslate">
@@ -446,6 +455,15 @@
</code>
</a>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.ipc.get_sender_ctx">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ get_sender_ctx()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.ipc.get_worker_ctx">
<code class="docutils literal notranslate">
@@ -455,6 +473,15 @@
</code>
</a>
</li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#datafusion.ipc.set_sender_ctx">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ set_sender_ctx()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#datafusion.ipc.set_worker_ctx">
<code class="docutils literal notranslate">
@@ -489,7 +516,7 @@
<section id="module-datafusion.ipc">
<span id="datafusion-ipc"></span><h1>datafusion.ipc<a class="headerlink" href="#module-datafusion.ipc" title="Link to this heading">¶</a></h1>
-<p>Worker-side setup for distributing DataFusion expressions.</p>
+<p>Driver- and worker-side setup for distributing DataFusion expressions.</p>
<p>When a <code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code> is shipped to a worker process (e.g. through
<code class="xref py py-func docutils literal notranslate"><span class="pre">multiprocessing.Pool()</span></code> or a Ray actor), the worker reconstructs the
expression against a <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>. If the expression references
@@ -524,17 +551,54 @@ payloads are not portable across Python minor versions. See
<a class="reference internal" href="../index.html#datafusion.Expr.to_bytes" title="datafusion.Expr.to_bytes"><code class="xref py py-meth docutils literal notranslate"><span class="pre">datafusion.Expr.to_bytes()</span></code></a> for examples of what travels by
value vs. by reference.</p>
</div>
+<p>On the driver side, call <a class="reference internal" href="#datafusion.ipc.set_sender_ctx" title="datafusion.ipc.set_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">set_sender_ctx()</span></code></a> to control how
+<code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code> encodes expressions — for example, to apply
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> to every pickled
+expression on this thread:</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">pickle</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">SessionContext</span><span class="p">,</span> <span class="n">col</span><span class="p">,</span> <span class="n">lit</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion.ipc</span><span class="w"> </span><span class="kn">import</span> <span class="n">clear_sender_ctx</span><span class="p">,</span> <span class="n">set_sender_ctx</span>
+<span class="gp">>>> </span><span class="n">driver_ctx</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span><span class="o">.</span><span class="n">with_python_udf_inlining</span><span class="p">(</span><span class="n">enabled</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">set_sender_ctx</span><span class="p">(</span><span class="n">driver_ctx</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="k">try</span><span class="p">:</span>
+<span class="gp">... </span> <span class="n">blob</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">col</span><span class="p">(</span><span class="s2">"a"</span><span class="p">)</span> <span class="o">+</span> <span class="n">lit</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
+<span class="gp">... </span><span class="k">finally</span><span class="p">:</span>
+<span class="gp">... </span> <span class="n">clear_sender_ctx</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="nb">isinstance</span><span class="p">(</span><span class="n">blob</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">)</span>
+<span class="go">True</span>
+</pre></div>
+</div>
+<p>Without a sender context the default codec is used (Python UDF
+inlining on). The sender context only affects pickle / <code class="docutils literal notranslate"><span class="pre">to_bytes</span></code>
+encoding; explicit <code class="docutils literal notranslate"><span class="pre">expr.to_bytes(ctx)</span></code> calls still use the supplied
+<code class="docutils literal notranslate"><span class="pre">ctx</span></code>.</p>
+<p>The thread-local sender context holds a strong reference to the
+installed <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> until <a class="reference internal" href="#datafusion.ipc.clear_sender_ctx" title="datafusion.ipc.clear_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">clear_sender_ctx()</span></code></a> is
+called or the thread exits. Long-running driver threads that install a sender
+context once and never clear it will retain that session for the
+lifetime of the thread; pair <a class="reference internal" href="#datafusion.ipc.set_sender_ctx" title="datafusion.ipc.set_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">set_sender_ctx()</span></code></a> with
+<a class="reference internal" href="#datafusion.ipc.clear_sender_ctx" title="datafusion.ipc.clear_sender_ctx"><code class="xref py py-func docutils literal notranslate"><span class="pre">clear_sender_ctx()</span></code></a> (e.g. in a <code class="docutils literal notranslate"><span class="pre">try</span></code>/<code class="docutils literal notranslate"><span class="pre">finally</span></code>) when the
+sender context is only needed for a bounded scope.</p>
<section id="functions">
<h2>Functions<a class="headerlink" href="#functions" title="Link to this heading">¶</a></h2>
<table class="autosummary longtable table autosummary">
<tbody>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.ipc.clear_worker_ctx" title="datafusion.ipc.clear_worker_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">clear_worker_ctx</span></code></a>(→ None)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.ipc.clear_sender_ctx" title="datafusion.ipc.clear_sender_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">clear_sender_ctx</span></code></a>(→ None)</p></td>
+<td><p>Remove this driver's installed sender <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.ipc.clear_worker_ctx" title="datafusion.ipc.clear_worker_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">clear_worker_ctx</span></code></a>(→ None)</p></td>
<td><p>Remove this worker's installed <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>.</p></td>
</tr>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.ipc.get_sender_ctx" title="datafusion.ipc.get_sender_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">get_sender_ctx</span></code></a>(→ datafusion.context.SessionContext | None)</p></td>
+<td><p>Return this driver's installed sender <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>, or <code class="docutils literal notranslate"><span class="pre">None</span></code>.</p></td>
+</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.ipc.get_worker_ctx" title="datafusion.ipc.get_worker_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">get_worker_ctx</span></code></a>(→ datafusion.context.SessionContext | None)</p></td>
<td><p>Return this worker's installed <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>, or <code class="docutils literal notranslate"><span class="pre">None</span></code>.</p></td>
</tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.ipc.set_worker_ctx" title="datafusion.ipc.set_worker_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">set_worker_ctx</span></code></a>(→ None)</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="#datafusion.ipc.set_sender_ctx" title="datafusion.ipc.set_sender_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">set_sender_ctx</span></code></a>(→ None)</p></td>
+<td><p>Install this driver's <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> for outbound pickles.</p></td>
+</tr>
+<tr class="row-even"><td><p><a class="reference internal" href="#datafusion.ipc.set_worker_ctx" title="datafusion.ipc.set_worker_ctx"><code class="xref py py-obj docutils literal notranslate"><span class="pre">set_worker_ctx</span></code></a>(→ None)</p></td>
<td><p>Install this worker's <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> for shipped expressions.</p></td>
</tr>
</tbody>
@@ -542,6 +606,25 @@ value vs. by reference.</p>
</section>
<section id="module-contents">
<h2>Module Contents<a class="headerlink" href="#module-contents" title="Link to this heading">¶</a></h2>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.ipc.clear_sender_ctx">
+<span class="sig-prename descclassname"><span class="pre">datafusion.ipc.</span></span><span class="sig-name descname"><span class="pre">clear_sender_ctx</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="headerlink" href="#datafusion.ipc.clear_sender_ctx" title="Link to this definition">¶</a></dt>
+<dd><p>Remove this driver’s installed sender <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>.</p>
+<p>After clearing, pickled expressions fall back to the default codec
+(Python UDF inlining on).</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">SessionContext</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion.ipc</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
+<span class="gp">... </span> <span class="n">set_sender_ctx</span><span class="p">,</span> <span class="n">clear_sender_ctx</span><span class="p">,</span> <span class="n">get_sender_ctx</span><span class="p">,</span>
+<span class="gp">... </span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">set_sender_ctx</span><span class="p">(</span><span class="n">SessionContext</span><span class="p">())</span>
+<span class="gp">>>> </span><span class="n">clear_sender_ctx</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">get_sender_ctx</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span>
+<span class="go">True</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.ipc.clear_worker_ctx">
<span class="sig-prename descclassname"><span class="pre">datafusion.ipc.</span></span><span class="sig-name descname"><span class="pre">clear_worker_ctx</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><span class="pre">None</span></span></span><a class="headerlink" href="#datafusion.ipc.clear_worker_ctx" title="Link to this definition">¶</a></dt>
@@ -561,6 +644,19 @@ capsule protocol must be registered on the global context to resolve.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.ipc.get_sender_ctx">
+<span class="sig-prename descclassname"><span class="pre">datafusion.ipc.</span></span><span class="sig-name descname"><span class="pre">get_sender_ctx</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><span class="pre">data [...]
+<dd><p>Return this driver’s installed sender <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code>, or <code class="docutils literal notranslate"><span class="pre">None</span></code>.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion.ipc</span><span class="w"> </span><span class="kn">import</span> <span class="n">get_sender_ctx</span><span class="p">,</span> <span class="n">clear_sender_ctx</span>
+<span class="gp">>>> </span><span class="n">clear_sender_ctx</span><span class="p">()</span>
+<span class="gp">>>> </span><span class="n">get_sender_ctx</span><span class="p">()</span> <span class="ow">is</span> <span class="kc">None</span>
+<span class="go">True</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.ipc.get_worker_ctx">
<span class="sig-prename descclassname"><span class="pre">datafusion.ipc.</span></span><span class="sig-name descname"><span class="pre">get_worker_ctx</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">→</span> <span class="sig-return-typehint"><a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext"><span class="pre">data [...]
@@ -574,6 +670,31 @@ capsule protocol must be registered on the global context to resolve.</p>
</div>
</dd></dl>
+<dl class="py function">
+<dt class="sig sig-object py" id="datafusion.ipc.set_sender_ctx">
+<span class="sig-prename descclassname"><span class="pre">datafusion.ipc.</span></span><span class="sig-name descname"><span class="pre">set_sender_ctx</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">ctx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext [...]
+<dd><p>Install this driver’s <code class="xref py py-class docutils literal notranslate"><span class="pre">SessionContext</span></code> for outbound pickles.</p>
+<p>Controls how <code class="xref py py-func docutils literal notranslate"><span class="pre">pickle.dumps()</span></code> encodes <code class="xref py py-class docutils literal notranslate"><span class="pre">Expr</span></code> instances on
+this thread. The most useful application is propagating a session
+configured with
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">SessionContext.with_python_udf_inlining()</span></code> so the toggle takes
+effect through pickle (which otherwise calls
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">Expr.to_bytes()</span></code> with no context and uses the default codec).</p>
+<p>Idempotent: overwrites any previous value. Stored in a thread-local
+slot, so worker threads on the driver may install their own contexts.
+Does not affect <code class="xref py py-meth docutils literal notranslate"><span class="pre">Expr.to_bytes()</span></code> calls that pass an explicit
+<code class="docutils literal notranslate"><span class="pre">ctx</span></code> — those continue to use the supplied context.</p>
+<p class="rubric">Examples</p>
+<div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion</span><span class="w"> </span><span class="kn">import</span> <span class="n">SessionContext</span>
+<span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">datafusion.ipc</span><span class="w"> </span><span class="kn">import</span> <span class="n">set_sender_ctx</span><span class="p">,</span> <span class="n">get_sender_ctx</span>
+<span class="gp">>>> </span><span class="n">driver</span> <span class="o">=</span> <span class="n">SessionContext</span><span class="p">()</span><span class="o">.</span><span class="n">with_python_udf_inlining</span><span class="p">(</span><span class="n">enabled</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">set_sender_ctx</span><span class="p">(</span><span class="n">driver</span><span class="p">)</span>
+<span class="gp">>>> </span><span class="n">get_sender_ctx</span><span class="p">()</span> <span class="ow">is</span> <span class="n">driver</span>
+<span class="go">True</span>
+</pre></div>
+</div>
+</dd></dl>
+
<dl class="py function">
<dt class="sig sig-object py" id="datafusion.ipc.set_worker_ctx">
<span class="sig-prename descclassname"><span class="pre">datafusion.ipc.</span></span><span class="sig-name descname"><span class="pre">set_worker_ctx</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">ctx</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="../context/index.html#datafusion.context.SessionContext" title="datafusion.context.SessionContext [...]
diff --git a/genindex.html b/genindex.html
index 313a2852..712489f8 100644
--- a/genindex.html
+++ b/genindex.html
@@ -1440,6 +1440,8 @@
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.chr">(in module datafusion.functions)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.clear_sender_ctx">clear_sender_ctx() (in module datafusion.ipc)</a>
+</li>
<li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.clear_worker_ctx">clear_worker_ctx() (in module datafusion.ipc)</a>
</li>
<li><a href="autoapi/datafusion/functions/index.html#datafusion.functions.coalesce">coalesce() (in module datafusion.functions)</a>
@@ -2296,6 +2298,8 @@
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.WindowFrameBound.get_offset">get_offset() (datafusion.expr.WindowFrameBound method)</a>
</li>
<li><a href="autoapi/datafusion/user_defined/index.html#datafusion.user_defined.WindowEvaluator.get_range">get_range() (datafusion.user_defined.WindowEvaluator method)</a>
+</li>
+ <li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.get_sender_ctx">get_sender_ctx() (in module datafusion.ipc)</a>
</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.WindowFrame.get_upper_bound">get_upper_bound() (datafusion.expr.WindowFrame method)</a>
@@ -3496,6 +3500,8 @@
<li><a href="autoapi/datafusion/dataframe_formatter/index.html#datafusion.dataframe_formatter.set_formatter">(in module datafusion.dataframe_formatter)</a>
</li>
</ul></li>
+ <li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.set_sender_ctx">set_sender_ctx() (in module datafusion.ipc)</a>
+</li>
<li><a href="autoapi/datafusion/ipc/index.html#datafusion.ipc.set_worker_ctx">set_worker_ctx() (in module datafusion.ipc)</a>
</li>
<li><a href="autoapi/datafusion/expr/index.html#datafusion.expr.SetVariable">SetVariable (in module datafusion.expr)</a>
@@ -4228,14 +4234,14 @@
<li><a href="autoapi/datafusion/options/index.html#datafusion.options.CsvReadOptions.with_file_sort_order">(datafusion.options.CsvReadOptions method)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.RuntimeEnvBuilder.with_greedy_memory_pool">with_greedy_memory_pool() (datafusion.context.RuntimeEnvBuilder method)</a>
<ul>
<li><a href="autoapi/datafusion/index.html#datafusion.RuntimeEnvBuilder.with_greedy_memory_pool">(datafusion.RuntimeEnvBuilder method)</a>
</li>
</ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a href="autoapi/datafusion/index.html#datafusion.CsvReadOptions.with_has_header">with_has_header() (datafusion.CsvReadOptions method)</a>
<ul>
@@ -4271,6 +4277,8 @@
<li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.with_physical_extension_codec">with_physical_extension_codec() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/unparser/index.html#datafusion.unparser.Unparser.with_pretty">with_pretty() (datafusion.unparser.Unparser method)</a>
+</li>
+ <li><a href="autoapi/datafusion/context/index.html#datafusion.context.SessionContext.with_python_udf_inlining">with_python_udf_inlining() (datafusion.context.SessionContext method)</a>
</li>
<li><a href="autoapi/datafusion/index.html#datafusion.CsvReadOptions.with_quote">with_quote() (datafusion.CsvReadOptions method)</a>
diff --git a/objects.inv b/objects.inv
index d9454405..e69bb5d2 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/searchindex.js b/searchindex.js
index 013aa046..80a1a46e 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[19, null], [37, "api-reference"]], "Additional Resources": [[39, "additional-resources"]], "Aggregate Functions": [[25, "aggregate-functions"], [32, "aggregate-functions"], [34, "aggregate-functions"]], "Aggregated vs Per-Partition Metrics": [[37, "aggregated-vs-per-partition-metrics"]], "Aggregation": [[25, null]], "Alternative Approach": [[20, "alternative-approach"]], "Apache Iceberg": [[36, "apache-iceberg"]], "Arrays": [[27, "arrays" [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[19, null], [37, "api-reference"]], "Additional Resources": [[39, "additional-resources"]], "Aggregate Functions": [[25, "aggregate-functions"], [32, "aggregate-functions"], [34, "aggregate-functions"]], "Aggregated vs Per-Partition Metrics": [[37, "aggregated-vs-per-partition-metrics"]], "Aggregation": [[25, null]], "Alternative Approach": [[20, "alternative-approach"]], "Apache Iceberg": [[36, "apache-iceberg"]], "Arrays": [[27, "arrays" [...]
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html b/user-guide/common-operations/aggregations.html
index 6de12df6..2cc3a4a2 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -662,15 +662,15 @@ Pokemon set. Since there will be many entries of <code class="docutils literal n
<span class="o">+----------+--------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+--------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Water</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Ice</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Water</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Dragon</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span class="p">,</span> <span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Dragon</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fighting</span> <span class="o">|</span> <span class="p">[]</span> <span class="o">|</span>
<span class="o">+----------+--------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
@@ -686,14 +686,14 @@ entries entirely removed. The second is we can use the <code class="docutils lit
<span class="o">+--------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+--------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="n">Water</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Fighting</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Water</span><span class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Dragon</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span class="p">]</span> <span class="o">|</span>
<span class="o">+--------+------------------------------------------------+</span>
@@ -705,15 +705,15 @@ entries entirely removed. The second is we can use the <code class="docutils lit
<span class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">Type</span> <span class="mi">2</span> <span class="n">List</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Water</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span class="n">Dark</span><span class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Water</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Dragon</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Grass</span> <span class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span class="p">,</span> <span class="n">Poison</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Dragon</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Ground</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span class="p">,</span> <span class="n">Fairy</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span class="p">,</span> <span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Fighting</span> <span class="o">|</span> <span class="o">|</span>
<span class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
diff --git a/user-guide/common-operations/basic-info.html b/user-guide/common-operations/basic-info.html
index 3b0b0b9d..5d008409 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -466,11 +466,11 @@
<span class="o">+-----+---------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">439</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">401</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">268</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">563</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">690</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">166</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">248</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span> <span class="n">java</span> <span class="o">|</span> <span class="mi">621</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span> <span class="n">haskell</span> <span class="o">|</span> <span class="mi">574</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mi">52</span> <span class="o">|</span> <span class="n">B</span> <span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
</pre></div>
</div>
@@ -481,8 +481,8 @@
<span class="o">+-----+--------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span class="n">random</span> <span class="o">|</span> <span class="n">groups</span> <span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">439</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">401</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span> <span class="n">python</span> <span class="o">|</span> <span class="mi">166</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mi">248</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
</pre></div>
</div>
@@ -500,11 +500,11 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">df</span><span class="o">.</span><span class="n">to_pandas</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span>
<span class="n">nrs</span> <span class="n">names</span> <span class="n">random</span> <span class="n">groups</span>
-<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">439</span> <span class="n">A</span>
-<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">401</span> <span class="n">A</span>
-<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">268</span> <span class="n">B</span>
-<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">563</span> <span class="n">C</span>
-<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">690</span> <span class="n">B</span>
+<span class="mi">0</span> <span class="mi">1</span> <span class="n">python</span> <span class="mi">166</span> <span class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span class="n">ruby</span> <span class="mi">248</span> <span class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span class="n">java</span> <span class="mi">621</span> <span class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span class="n">haskell</span> <span class="mi">574</span> <span class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span class="n">go</span> <span class="mi">52</span> <span class="n">B</span>
</pre></div>
</div>
<p><a class="reference internal" href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe" title="datafusion.dataframe.DataFrame.describe"><code class="xref py py-func docutils literal notranslate"><span class="pre">describe()</span></code></a> shows a quick statistic summary of your data:</p>
@@ -516,11 +516,11 @@ passing them to an Arrow table, and then converting them to a pandas DataFrame.<
<span class="o">+------------+--------------------+-------+--------------------+--------+</span>
<span class="o">|</span> <span class="n">count</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">null_count</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span class="mf">0.0</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">472.2</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">160.89655061560518</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">268.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">690.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">439.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">332.2</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span class="o">|</span> <span class="mf">1.5811388300841898</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">252.53554205299497</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span class="o">|</span> <span class="mf">1.0</span> <span class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span class="mf">52.0</span> <span class="o">|</span> <span class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span class="o">|</span> <span class="mf">5.0</span> <span class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span class="mf">621.0</span> <span class="o">|</span> <span class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span class="o">|</span> <span class="mf">3.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span class="mf">248.0</span> <span class="o">|</span> <span class="n">null</span> <span class="o">|</span>
<span class="o">+------------+--------------------+-------+--------------------+--------+</span>
</pre></div>
</div>
diff --git a/user-guide/common-operations/functions.html b/user-guide/common-operations/functions.html
index ccfeea57..933f7687 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -576,16 +576,16 @@ In here we will cover some of the more popular use cases. If you want to view al
<span class="o">+-------------------------------+</span>
<span class="o">|</span> <span class="n">now</span><span class="p">()</span> <span class="o">|</span>
<span class="o">+-------------------------------+</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T17</span><span class="p">:</span><span class="mi">50</span><span class="p">:</span><span class="mf">30.812699989</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span class="o">-</span><span class="mi">05</span><span class="o">-</span><span class="mi">26</span><span class="n">T20</span><span class="p">:</span><span class="mi">01</span><span class="p">:</span><span class="mf">28.204558433</span> <span class="o">|</span>
<span class="o">+-------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span class="o">.</span>
</pre></div>
diff --git a/user-guide/common-operations/udf-and-udfa.html b/user-guide/common-operations/udf-and-udfa.html
index 8610418c..72e0993c 100644
--- a/user-guide/common-operations/udf-and-udfa.html
+++ b/user-guide/common-operations/udf-and-udfa.html
@@ -649,7 +649,7 @@ over column references and literals, so the optimizer can analyze it:</p>
<span class="n">In</span> <span class="p">[</span><span class="mi">29</span><span class="p">]:</span> <span class="nb">print</span><span class="p">(</span><span class="n">native_filtered</span><span class="o">.</span><span class="n">execution_plan</span><span class="p">()</span><span class="o">.</span><span class="n">display_indent</span><span class="p">())</span>
<span class="n">FilterExec</span><span class="p">:</span> <span class="n">brand</span><span class="o">@</span><span class="mi">1</span> <span class="o">=</span> <span class="n">A</span> <span class="n">AND</span> <span class="n">qty</span><span class="o">@</span><span class="mi">2</span> <span class="o">>=</span> <span class="mi">150</span>
<span class="n">RepartitionExec</span><span class="p">:</span> <span class="n">partitioning</span><span class="o">=</span><span class="n">RoundRobinBatch</span><span class="p">(</span><span class="mi">4</span><span class="p">),</span> <span class="n">input_partitions</span><span class="o">=</span><span class="mi">1</span>
- <span class="n">DataSourceExec</span><span class="p">:</span> <span class="n">file_groups</span><span class="o">=</span><span class="p">{</span><span class="mi">1</span> <span class="n">group</span><span class="p">:</span> <span class="p">[[</span><span class="n">tmp</span><span class="o">/</span><span class="n">tmp9e54s1q1</span><span class="o">/</span><span class="n">items</span><span class="o">.</span><span class="n">parquet</span><span class="p">]]},</span> <span class="n">projec [...]
+ <span class="n">DataSourceExec</span><span class="p">:</span> <span class="n">file_groups</span><span class="o">=</span><span class="p">{</span><span class="mi">1</span> <span class="n">group</span><span class="p">:</span> <span class="p">[[</span><span class="n">tmp</span><span class="o">/</span><span class="n">tmp5h8pif9w</span><span class="o">/</span><span class="n">items</span><span class="o">.</span><span class="n">parquet</span><span class="p">]]},</span> <span class="n">projec [...]
</pre></div>
</div>
<p>Notice the <code class="docutils literal notranslate"><span class="pre">DataSourceExec</span></code> line. It carries three annotations the
@@ -682,7 +682,7 @@ bloom filter or dictionary is available to skip pages.</p></li>
<span class="n">In</span> <span class="p">[</span><span class="mi">33</span><span class="p">]:</span> <span class="nb">print</span><span class="p">(</span><span class="n">udf_filtered</span><span class="o">.</span><span class="n">execution_plan</span><span class="p">()</span><span class="o">.</span><span class="n">display_indent</span><span class="p">())</span>
<span class="n">FilterExec</span><span class="p">:</span> <span class="n">brand_qty_filter</span><span class="p">(</span><span class="n">CAST</span><span class="p">(</span><span class="n">brand</span><span class="o">@</span><span class="mi">1</span> <span class="n">AS</span> <span class="n">Utf8</span><span class="p">),</span> <span class="n">qty</span><span class="o">@</span><span class="mi">2</span><span class="p">)</span>
<span class="n">RepartitionExec</span><span class="p">:</span> <span class="n">partitioning</span><span class="o">=</span><span class="n">RoundRobinBatch</span><span class="p">(</span><span class="mi">4</span><span class="p">),</span> <span class="n">input_partitions</span><span class="o">=</span><span class="mi">1</span>
- <span class="n">DataSourceExec</span><span class="p">:</span> <span class="n">file_groups</span><span class="o">=</span><span class="p">{</span><span class="mi">1</span> <span class="n">group</span><span class="p">:</span> <span class="p">[[</span><span class="n">tmp</span><span class="o">/</span><span class="n">tmp9e54s1q1</span><span class="o">/</span><span class="n">items</span><span class="o">.</span><span class="n">parquet</span><span class="p">]]},</span> <span class="n">projec [...]
+ <span class="n">DataSourceExec</span><span class="p">:</span> <span class="n">file_groups</span><span class="o">=</span><span class="p">{</span><span class="mi">1</span> <span class="n">group</span><span class="p">:</span> <span class="p">[[</span><span class="n">tmp</span><span class="o">/</span><span class="n">tmp5h8pif9w</span><span class="o">/</span><span class="n">items</span><span class="o">.</span><span class="n">parquet</span><span class="p">]]},</span> <span class="n">projec [...]
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">DataSourceExec</span></code> now carries only <code class="docutils literal notranslate"><span class="pre">predicate=brand_qty_filter(...)</span></code>.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org