You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/14 23:41:15 UTC
(datafusion) branch asf-site updated: Publish built docs triggered by e732212cabd030d500efc35049448b3714017c7b
This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cf5da28571 Publish built docs triggered by e732212cabd030d500efc35049448b3714017c7b
cf5da28571 is described below
commit cf5da2857154d26aeed15f680c8586d176690696
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Oct 14 23:41:09 2026 +0000
Publish built docs triggered by e732212cabd030d500efc35049448b3714017c7b
---
.../library-user-guide/upgrading/56.0.0.md.txt | 86 ++++++++++++++++++++++
library-user-guide/upgrading/56.0.0.html | 77 +++++++++++++++++++
searchindex.js | 2 +-
3 files changed, 164 insertions(+), 1 deletion(-)
diff --git a/_sources/library-user-guide/upgrading/56.0.0.md.txt b/_sources/library-user-guide/upgrading/56.0.0.md.txt
index b344eff617..cb373fdc09 100644
--- a/_sources/library-user-guide/upgrading/56.0.0.md.txt
+++ b/_sources/library-user-guide/upgrading/56.0.0.md.txt
@@ -1076,3 +1076,89 @@ let plan = provider
.await?
.into_inner();
```
+
+### Extension `ExecutionPlan`s can decode without a `PhysicalExtensionCodec`
+
+`ExecutionPlan::try_to_proto` let a plan serialize itself, but there was no way back: decoding an extension node always routed through `PhysicalExtensionCodec::try_decode`. `PhysicalExtensionNode` carried no type discriminator, so the codec _was_ the discriminator. `ComposedPhysicalExtensionCodec` works around that by writing the position of the encoding codec into the payload, which means the decoding side must register the same codecs in the same order, and a name collision between two [...]
+
+`PhysicalExtensionNode` now has an optional `plan_name`, and a session can map that name to a decoder. An extension plan implements `ExtensionPlanFromProto`, which pairs the name it is dispatched by with the constructor that rebuilds it, and is registered on the session that will decode it:
+
+```rust,ignore
+use datafusion::physical_plan::proto::{
+ ExecutionPlanDecodeCtx, ExecutionPlanEncodeCtx, ExtensionPlanFromProto,
+ ProtoDecoderRegistry, register_execution_plan,
+};
+
+impl ExecutionPlan for MyExec {
+ // ...
+ fn try_to_proto(
+ &self,
+ ctx: &ExecutionPlanEncodeCtx<'_>,
+ ) -> Result<Option<PhysicalPlanNode>> {
+ // `extension_node` stamps `Self::NAME` on the node, so the name on the
+ // wire and the registry key cannot drift apart.
+ Ok(Some(ctx.extension_node::<Self>(
+ my_payload_bytes(self)?,
+ self.children(),
+ )?))
+ }
+
+}
+
+impl ExtensionPlanFromProto for MyExec {
+ // Namespace the name so a collision with another crate is an error at
+ // registration rather than a wrong decode.
+ const NAME: &'static str = "my-crate.MyExec";
+
+ fn try_from_proto(
+ node: &PhysicalPlanNode,
+ ctx: &ExecutionPlanDecodeCtx<'_>,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ let extension = expect_plan_variant!(node, PhysicalPlanType::Extension, "Extension");
+ let children = ctx.decode_children(&extension.inputs)?;
+ // `ctx.task_ctx()` is available here, so a plan that rebuilds session
+ // state at decode time (a connection pool, say) can do so.
+ my_plan_from_bytes(&extension.node, children)
+ }
+}
+
+// On every session that decodes the plan — for a distributed engine, the
+// workers as well as the coordinator.
+let mut registry = ProtoDecoderRegistry::new();
+register_execution_plan::<MyExec>(&mut registry)?;
+let config = SessionConfig::new().with_extension(Arc::new(registry));
+```
+
+Only extension plans implement `ExtensionPlanFromProto`, and `register_execution_plan` takes that trait as its bound. A built-in plan has a wire variant of its own and keeps its inherent `try_from_proto`, so `register_execution_plan::<FilterExec>(&mut registry)` does not compile.
+
+`ProtoDecoderRegistry` is shared by every kind of extension DataFusion can decode, not only plans. Its key is the trait plus the name, so a plan and an expression may use the same name, and one session carries one registry rather than one per kind.
+
+The application that owns the session builds the registry. A session carries at most one, and `with_extension` replaces what is there, so a library must not attach a registry of its own: it would silently discard another library's. A library exposes a function that fills a registry it is handed, and the application composes them:
+
+```rust,ignore
+// In each library that ships extension types:
+pub fn register(registry: &mut ProtoDecoderRegistry) -> Result<()> {
+ register_execution_plan::<MyExec>(registry)
+}
+
+// In the application that owns the session:
+let mut registry = ProtoDecoderRegistry::new();
+shuffle_lib::register(&mut registry)?;
+sampling_lib::register(&mut registry)?;
+let config = SessionConfig::new().with_extension(Arc::new(registry));
+```
+
+The libraries need to know nothing about each other, and the order they are called in does not matter. Dispatch is by name, and a real name collision is an error from `register_execution_plan` rather than a wrong decode later.
+
+A registered plan needs no `PhysicalExtensionCodec` at all, and two crates' plans coexist on one session without a `ComposedPhysicalExtensionCodec`.
+
+**Who is affected:**
+
+- Nobody is required to change anything. This is opt-in per plan type: a `PhysicalExtensionNode` with no name, or with a name the decoding session does not know, takes the `PhysicalExtensionCodec` chain exactly as before. On the binary wire format the new field is additive: old writers omit it and old readers skip it.
+- The generated JSON codec is stricter than the binary one. `pbjson` rejects a field it does not know, so a named node written by a new writer cannot be read as JSON by a reader on an older DataFusion. A plan that does not opt in writes no name and is unaffected. If you serialize plans as JSON across versions, upgrade the readers before you opt a plan in.
+- Users constructing `PhysicalExtensionNode` with an exhaustive struct literal must set the new `plan_name` field (`None` reproduces the previous behavior).
+- Replacing a `PhysicalExtensionCodec` with a registration changes only _how_ the payload is dispatched, not what the payload is. Readers running an older DataFusion ignore `plan_name` and hand the payload to their codec, so a mixed fleet keeps working only while the bytes your `try_to_proto` writes stay decodable by the codec you removed.
+
+`PhysicalExtensionCodec` is not deprecated: extension `PhysicalExpr`s and UDF payloads still need it, and it remains the fallback for unmigrated plans.
+
+See [issue #24625](https://github.com/apache/datafusion/issues/24625) for details.
diff --git a/library-user-guide/upgrading/56.0.0.html b/library-user-guide/upgrading/56.0.0.html
index f24d542896..d7a02eda61 100644
--- a/library-user-guide/upgrading/56.0.0.html
+++ b/library-user-guide/upgrading/56.0.0.html
@@ -1394,6 +1394,82 @@ the API breaks.</p>
</div>
</section>
</section>
+<section id="extension-executionplans-can-decode-without-a-physicalextensioncodec">
+<h3>Extension <code class="docutils literal notranslate"><span class="pre">ExecutionPlan</span></code>s can decode without a <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec</span></code><a class="headerlink" href="#extension-executionplans-can-decode-without-a-physicalextensioncodec" title="Link to this heading">#</a></h3>
+<p><code class="docutils literal notranslate"><span class="pre">ExecutionPlan::try_to_proto</span></code> let a plan serialize itself, but there was no way back: decoding an extension node always routed through <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec::try_decode</span></code>. <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionNode</span></code> carried no type discriminator, so the codec <em>was</em> the discriminato [...]
+<p><code class="docutils literal notranslate"><span class="pre">PhysicalExtensionNode</span></code> now has an optional <code class="docutils literal notranslate"><span class="pre">plan_name</span></code>, and a session can map that name to a decoder. An extension plan implements <code class="docutils literal notranslate"><span class="pre">ExtensionPlanFromProto</span></code>, which pairs the name it is dispatched by with the constructor that rebuilds it, and is registered on the session [...]
+<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="k">use</span><span class="w"> </span><span class="n">datafusion</span><span class="p">::</span><span class="n">physical_plan</span><span class="p">::</span><span class="n">proto</span><span class="p">::{</span>
+<span class="w"> </span><span class="n">ExecutionPlanDecodeCtx</span><span class="p">,</span><span class="w"> </span><span class="n">ExecutionPlanEncodeCtx</span><span class="p">,</span><span class="w"> </span><span class="n">ExtensionPlanFromProto</span><span class="p">,</span>
+<span class="w"> </span><span class="n">ProtoDecoderRegistry</span><span class="p">,</span><span class="w"> </span><span class="n">register_execution_plan</span><span class="p">,</span>
+<span class="p">};</span>
+
+<span class="k">impl</span><span class="w"> </span><span class="n">ExecutionPlan</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyExec</span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="c1">// ...</span>
+<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">try_to_proto</span><span class="p">(</span>
+<span class="w"> </span><span class="o">&</span><span class="bp">self</span><span class="p">,</span>
+<span class="w"> </span><span class="n">ctx</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">ExecutionPlanEncodeCtx</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span><span class="p">,</span>
+<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="nb">Option</span><span class="o"><</span><span class="n">PhysicalPlanNode</span><span class="o">>></span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="c1">// `extension_node` stamps `Self::NAME` on the node, so the name on the</span>
+<span class="w"> </span><span class="c1">// wire and the registry key cannot drift apart.</span>
+<span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="nb">Some</span><span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="n">extension_node</span><span class="p">::</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">(</span>
+<span class="w"> </span><span class="n">my_payload_bytes</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">?</span><span class="p">,</span>
+<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">children</span><span class="p">(),</span>
+<span class="w"> </span><span class="p">)</span><span class="o">?</span><span class="p">))</span>
+<span class="w"> </span><span class="p">}</span>
+
+<span class="p">}</span>
+
+<span class="k">impl</span><span class="w"> </span><span class="n">ExtensionPlanFromProto</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyExec</span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="c1">// Namespace the name so a collision with another crate is an error at</span>
+<span class="w"> </span><span class="c1">// registration rather than a wrong decode.</span>
+<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="n">NAME</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="nb">static</span><span class="w"> </span><span class="kt">str</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"my-crate.MyExec"</span><span class="p">;</span>
+
+<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">try_from_proto</span><span class="p">(</span>
+<span class="w"> </span><span class="n">node</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">PhysicalPlanNode</span><span class="p">,</span>
+<span class="w"> </span><span class="n">ctx</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">ExecutionPlanDecodeCtx</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span><span class="p">,</span>
+<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="n">Arc</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">ExecutionPlan</span><span class="o">>></span><span class="w"> </span><span class="p">{</span>
+<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">extension</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">expect_plan_variant</span><span class="o">!</span><span class="p">(</span><span class="n">node</span><span class="p">,</span><span class="w"> </span><span class="n">PhysicalPlanType</span><span class="p">::</span><span class="n">Extension</span><span class="p">,</span><span class="w"> </sp [...]
+<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">children</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ctx</span><span class="p">.</span><span class="n">decode_children</span><span class="p">(</span><span class="o">&</span><span class="n">extension</span><span class="p">.</span><span class="n">inputs</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
+<span class="w"> </span><span class="c1">// `ctx.task_ctx()` is available here, so a plan that rebuilds session</span>
+<span class="w"> </span><span class="c1">// state at decode time (a connection pool, say) can do so.</span>
+<span class="w"> </span><span class="n">my_plan_from_bytes</span><span class="p">(</span><span class="o">&</span><span class="n">extension</span><span class="p">.</span><span class="n">node</span><span class="p">,</span><span class="w"> </span><span class="n">children</span><span class="p">)</span>
+<span class="w"> </span><span class="p">}</span>
+<span class="p">}</span>
+
+<span class="c1">// On every session that decodes the plan — for a distributed engine, the</span>
+<span class="c1">// workers as well as the coordinator.</span>
+<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">registry</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ProtoDecoderRegistry</span><span class="p">::</span><span class="n">new</span><span class="p">();</span>
+<span class="n">register_execution_plan</span><span class="p">::</span><span class="o"><</span><span class="n">MyExec</span><span class="o">></span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">registry</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
+<span class="kd">let</span><span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SessionConfig</span><span class="p">::</span><span class="n">new</span><span class="p">().</span><span class="n">with_extension</span><span class="p">(</span><span class="n">Arc</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">registry</span><span class="p">));</span>
+</pre></div>
+</div>
+<p>Only extension plans implement <code class="docutils literal notranslate"><span class="pre">ExtensionPlanFromProto</span></code>, and <code class="docutils literal notranslate"><span class="pre">register_execution_plan</span></code> takes that trait as its bound. A built-in plan has a wire variant of its own and keeps its inherent <code class="docutils literal notranslate"><span class="pre">try_from_proto</span></code>, so <code class="docutils literal notranslate"><span class="pre">r [...]
+<p><code class="docutils literal notranslate"><span class="pre">ProtoDecoderRegistry</span></code> is shared by every kind of extension DataFusion can decode, not only plans. Its key is the trait plus the name, so a plan and an expression may use the same name, and one session carries one registry rather than one per kind.</p>
+<p>The application that owns the session builds the registry. A session carries at most one, and <code class="docutils literal notranslate"><span class="pre">with_extension</span></code> replaces what is there, so a library must not attach a registry of its own: it would silently discard another library’s. A library exposes a function that fills a registry it is handed, and the application composes them:</p>
+<div class="highlight-rust notranslate"><div class="highlight"><pre><span></span><span class="c1">// In each library that ships extension types:</span>
+<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">register</span><span class="p">(</span><span class="n">registry</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">ProtoDecoderRegistry</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span [...]
+<span class="w"> </span><span class="n">register_execution_plan</span><span class="p">::</span><span class="o"><</span><span class="n">MyExec</span><span class="o">></span><span class="p">(</span><span class="n">registry</span><span class="p">)</span>
+<span class="p">}</span>
+
+<span class="c1">// In the application that owns the session:</span>
+<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">registry</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ProtoDecoderRegistry</span><span class="p">::</span><span class="n">new</span><span class="p">();</span>
+<span class="n">shuffle_lib</span><span class="p">::</span><span class="n">register</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">registry</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
+<span class="n">sampling_lib</span><span class="p">::</span><span class="n">register</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">registry</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
+<span class="kd">let</span><span class="w"> </span><span class="n">config</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SessionConfig</span><span class="p">::</span><span class="n">new</span><span class="p">().</span><span class="n">with_extension</span><span class="p">(</span><span class="n">Arc</span><span class="p">::</span><span class="n">new</span><span class="p">(</span><span class="n">registry</span><span class="p">));</span>
+</pre></div>
+</div>
+<p>The libraries need to know nothing about each other, and the order they are called in does not matter. Dispatch is by name, and a real name collision is an error from <code class="docutils literal notranslate"><span class="pre">register_execution_plan</span></code> rather than a wrong decode later.</p>
+<p>A registered plan needs no <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec</span></code> at all, and two crates’ plans coexist on one session without a <code class="docutils literal notranslate"><span class="pre">ComposedPhysicalExtensionCodec</span></code>.</p>
+<p><strong>Who is affected:</strong></p>
+<ul class="simple">
+<li><p>Nobody is required to change anything. This is opt-in per plan type: a <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionNode</span></code> with no name, or with a name the decoding session does not know, takes the <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec</span></code> chain exactly as before. On the binary wire format the new field is additive: old writers omit it and old readers skip it.</p></li>
+<li><p>The generated JSON codec is stricter than the binary one. <code class="docutils literal notranslate"><span class="pre">pbjson</span></code> rejects a field it does not know, so a named node written by a new writer cannot be read as JSON by a reader on an older DataFusion. A plan that does not opt in writes no name and is unaffected. If you serialize plans as JSON across versions, upgrade the readers before you opt a plan in.</p></li>
+<li><p>Users constructing <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionNode</span></code> with an exhaustive struct literal must set the new <code class="docutils literal notranslate"><span class="pre">plan_name</span></code> field (<code class="docutils literal notranslate"><span class="pre">None</span></code> reproduces the previous behavior).</p></li>
+<li><p>Replacing a <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec</span></code> with a registration changes only <em>how</em> the payload is dispatched, not what the payload is. Readers running an older DataFusion ignore <code class="docutils literal notranslate"><span class="pre">plan_name</span></code> and hand the payload to their codec, so a mixed fleet keeps working only while the bytes your <code class="docutils literal notranslate"><span class= [...]
+</ul>
+<p><code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec</span></code> is not deprecated: extension <code class="docutils literal notranslate"><span class="pre">PhysicalExpr</span></code>s and UDF payloads still need it, and it remains the fallback for unmigrated plans.</p>
+<p>See <a class="reference external" href="https://github.com/apache/datafusion/issues/24625">issue #24625</a> for details.</p>
+</section>
</section>
</section>
@@ -1494,6 +1570,7 @@ the API breaks.</p>
<li class="toc-h4 nav-item toc-entry"><a class="reference internal nav-link" href="#example">Example</a></li>
</ul>
</li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#extension-executionplans-can-decode-without-a-physicalextensioncodec">Extension <code class="docutils literal notranslate"><span class="pre">ExecutionPlan</span></code>s can decode without a <code class="docutils literal notranslate"><span class="pre">PhysicalExtensionCodec</span></code></a></li>
</ul>
</li>
</ul>
diff --git a/searchindex.js b/searchindex.js
index 78fa043586..c5feaf1145 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"!=":[[79,"op-neq"]],"!~":[[79,"op-re-not-match"]],"!~*":[[79,"op-re-not-match-i"]],"!~~":[[79,"id19"]],"!~~*":[[79,"id20"]],"%":[[79,"op-modulo"]],"&":[[79,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[17,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[79,"op-multiply"]],"+":[[79,"op-plus"]],"-":[[79,"op-minus"]],"/":[[79,"op-divide"]],"1. Array Literal Construction":[[84,"array-l [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"!=":[[79,"op-neq"]],"!~":[[79,"op-re-not-match"]],"!~*":[[79,"op-re-not-match-i"]],"!~~":[[79,"id19"]],"!~~*":[[79,"id20"]],"%":[[79,"op-modulo"]],"&":[[79,"op-bit-and"]],"(relation, name) tuples in logical fields and logical columns are unique":[[17,"relation-name-tuples-in-logical-fields-and-logical-columns-are-unique"]],"*":[[79,"op-multiply"]],"+":[[79,"op-plus"]],"-":[[79,"op-minus"]],"/":[[79,"op-divide"]],"1. Array Literal Construction":[[84,"array-l [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org