You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/08/07 19:21:27 UTC

[PR] Make extension codecs composable [datafusion-python]

timsaucer opened a new pull request, #1678:
URL: https://github.com/apache/datafusion-python/pull/1678

   # Which issue does this PR close?
   
   Part 2 of 3 in the split of #1672 (part 1: FFI query planner support; part 3: atomic `SessionContext.with_extensions`). Stacked on part 1.
   
   # Rationale for this change
   
   Supporting a foreign planner surfaced a codec problem: a query can involve three independent native libraries (datafusion-python, a provider library, and a planner library), and each library needs its extension codecs active on the session at the same time. Previously, installing a logical or physical extension codec replaced the prior codec, so the second library's install silently discarded the first — plans then failed later with a confusing decode error.
   
   # What changes are included in this PR?
   
   - `with_logical_extension_codec` / `with_physical_extension_codec` now prepend to a codec chain instead of replacing the prior codec. The most recently installed codec is consulted first, falling through codec by codec to DataFusion's default codec. A codec signals "not mine" by returning an error.
   - Encoding runs each codec against a scratch buffer so failed attempts leave no partial bytes, and treats Ok-with-no-bytes (encode by name) as "no opinion" so later codecs still get a chance.
   - When every codec in the chain fails, the errors are aggregated so the owning codec's diagnostic is not masked by the default codec's generic error.
   - Fixed a latent bug where installing a codec silently reset `python_udf_inlining` back to enabled.
   - `docs/source/contributor-guide/ffi.md` gains sections on composable codecs: family-prefix discipline, and that registration order between libraries no longer matters.
   
   # Are there any user-facing changes?
   
   Behavior change: installing an extension codec now composes with previously installed codecs instead of replacing them. Code that relied on replacement semantics (installing a codec to remove a prior one) is affected; all other usage keeps working and no longer loses earlier codecs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#issuecomment-5496630092

   @ntjohnson1 on your question about seeing which codec triggers for a given call — partly answered now, though not in `EXPLAIN`.
   
   `SessionContext.logical_extension_codec_ids()` and `physical_extension_codec_ids()` list what's installed, in install order, and those ids are exactly what a payload carries — so a decode failure names the codec that wrote the bytes and lists what the session actually has, e.g. `a table provider was encoded by extension codec 'lib_a.Codec', which is not installed on this session (installed: lib_c.Codec)`.
   
   What's still missing is your actual question: per-call attribution, "which codec handled this node in this plan". Encoding decides that by walking the chain until one claims the object, and nothing records the winner. Surfacing it in `EXPLAIN` would mean threading it through plan formatting, which I'd rather not fold into this PR. Happy to file it as a follow-up if you think it's worth having.
   
   Your other three comments are addressed as well: the `inner()` accessor is replaced by `codecs()` / `codec_ids()` / `terminal()` on the Rust side and `logical_extension_codec_ids()` / `physical_extension_codec_ids()` from Python, and both the contributor guide and the example README now state that the terminal fallback is `Default{Logical,Physical}ExtensionCodec` unless a Rust caller supplies another to `Python{Logical,Physical}Codec::new`, which is not reachable from the Python surface — so a chain of `[lib_a, lib_b]` with no default behind it isn't something you can set up from Python.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3905881375


##########
crates/core/src/codec.rs:
##########
@@ -223,32 +224,129 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Run `f` against each codec in `chain`, returning the first `Ok`.
+///
+/// A codec signals "not mine" by returning an error, so the chain
+/// keeps trying until a codec succeeds. When every codec fails and the
+/// chain has more than one entry, the errors are aggregated into a
+/// single message — returning only the last error would surface the
+/// terminal `Default*ExtensionCodec` "not provided" message and mask
+/// the more specific diagnostic from an installed codec (e.g. a
+/// corrupt-token error from the codec that owns the payload family).
+fn chain_try<C: ?Sized, R>(chain: &[Arc<C>], what: &str, f: impl Fn(&C) -> Result<R>) -> Result<R> {
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {

Review Comment:
   You were both right, and I've reworked the dispatch rather than keeping trial decoding. Thanks for the #16980 / #16986 pointers — the CSV-decoded-as-Parquet case is exactly the failure mode, and it's not one a codec can defend against from inside, since `MyMessage::decode(buf)` has no prefix to check and cannot decline.
   
   Every payload an installed codec writes is now wrapped in an envelope naming its author: `DFPYCHN | version: u8 | id_len: u32 (LE) | id | blob`. Decoding reads the id and consults exactly one codec. Nothing is ever offered to a codec that did not write it, so structurally similar prost messages can't cross libraries. Codecs themselves are unchanged — they receive their payload byte for byte and never see the envelope.
   
   On reusing `ComposedPhysicalExtensionCodec`: I looked at it closely and it doesn't fit here, for three reasons.
   
   1. It keys on `encoder_position`, which is sound when both ends build the same list in the same order. That's true for the consumers it was written for — Ballista's list is a compile-time constant, and datafusion-distributed pins its own codec at index 0 and appends user codecs rebuilt from the same startup code on every node. It isn't true here: the chain is assembled by user Python, and `Expr.to_bytes(ctx1)` / `Expr.from_bytes(ctx2)` puts two independently configured sessions on either end of one payload. An index names a different codec in the decoder as soon as install order differs. `test_decode_survives_a_different_install_order` is the case.
   2. It only exists for the physical layer. There's no `ComposedLogicalExtensionCodec`, and this needs both — table providers and UDFs go through the logical codec.
   3. `encode_protobuf` always wraps in `DataEncoderTuple`, so a codec that writes zero bytes still emits a non-empty payload. That sets `fun_definition` and permanently skips the `FunctionRegistry` lookup the decoder does first, which breaks DataFusion's encode-by-name path. I need that path to keep working — `NameOnlyUdfCodec` in the FFI example owns functions that are fully described by their names and encodes nothing at all.
   
   The reasoning is recorded in the code at `crates/core/src/codec.rs` (see `ChainEntry` and `chain_decode`, which cite both upstream issues) so the next person doesn't re-derive it, and the "Composable codecs" section of `docs/source/contributor-guide/ffi.md` covers it for extension authors. The empty-payload case is the one place codecs are still consulted in turn, because there are no bytes to carry an identity; that question is "do you own the function named `x`", which is name-scoped, and two codecs disagreeing means they've already collided in the function registry.
   
   Coverage is in `examples/datafusion-ffi-example/python/tests/_test_logical_extension_codec.py`, across a real FFI boundary: `test_decode_dispatches_to_the_codec_that_encoded`, `test_decode_survives_a_different_install_order`, `test_decode_names_the_codec_that_is_not_installed`, `test_installing_a_codec_cannot_hijack_an_earlier_codecs_objects`, and `test_name_only_codec_round_trips_without_a_payload`. The PR description has been rewritten to describe this design — the earlier text described the trial-decode version you reviewed.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3786211112


##########
crates/core/src/codec.rs:
##########
@@ -443,22 +568,31 @@ fn refuse_inline_payload(kind: &str, name: &str) -> datafusion::error::DataFusio
 /// would round-trip at the logical level but break at the physical
 /// level. Both layers reuse the shared payload framing
 /// ([`PY_SCALAR_UDF_FAMILY`] et al.) so the wire format is identical.
-#[derive(Debug)]
+#[derive(Debug, Clone)]
 pub struct PythonPhysicalCodec {
-    inner: Arc<dyn PhysicalExtensionCodec>,
+    chain: Vec<Arc<dyn PhysicalExtensionCodec>>,
     python_udf_inlining: bool,
 }
 
 impl PythonPhysicalCodec {
     pub fn new(inner: Arc<dyn PhysicalExtensionCodec>) -> Self {
         Self {
-            inner,
+            chain: vec![inner],
             python_udf_inlining: true,
         }
     }
 
-    pub fn inner(&self) -> &Arc<dyn PhysicalExtensionCodec> {

Review Comment:
   Similar note about directly referencing codec chain



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer merged PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3935993010


##########
docs/source/contributor-guide/ffi.md:
##########
@@ -248,10 +248,96 @@ foreign planner. This lets the planner decode provider-owned objects and lets
 process-local tokens to demonstrate ownership; production codecs should serialize
 durable metadata instead.
 
-The current Python API has one external logical codec and one external physical codec.
-Installing another codec replaces the prior codec rather than composing a registry.
-The example therefore has one external codec owner, and the planner uses built-in
-physical nodes. Install the provider codecs before the planner where possible.
+### Composable codecs
+
+Extension codecs compose. Each call to `with_logical_extension_codec` or
+`with_physical_extension_codec` appends the codec to the session's codec chain
+rather than replacing prior codecs.
+
+**Nothing is asked of the codec itself.** Implement `LogicalExtensionCodec` or
+`PhysicalExtensionCodec` exactly as you would for a session that installs only
+yours. When your codec writes bytes into a serialized plan, datafusion-python
+records which codec wrote them, and strips that record off again before handing the
+bytes back. So your codec receives, byte for byte, the payload it wrote, and is
+never offered a payload another codec wrote.
+
+A codec that also ships to hosts which dispatch differently may still want its own
+guard against foreign payloads. Keeping one is fine; it is simply not needed for the
+datafusion-python path.
+
+That record is the codec's **id**: a short string stored inside the plan, naming the
+codec that wrote each payload. Because plans are decoded in another process — or
+another program — the id has to name the same codec there as it did where the plan
+was written.
+
+Ids are assigned for you. A codec's id is normally its exporting class's import
+path, such as `my_library.Codec`, which is what you will see in
+`logical_extension_codec_ids()` and in decode errors. You choose one yourself in
+three cases:
+
+- **Two instances of one class.** Both get the same id, so the second install
+  raises `ValueError`. Pass `codec_id=` to tell them apart.
+- **A bare `PyCapsule`.** A capsule has no class to take a name from, so it gets an
+  id private to the session that installed it. Plans it encodes fail with a clear
+  error on any other session, rather than being decoded by the wrong codec. Pass
+  `codec_id=` if those plans have to cross sessions.
+- **A class you intend to rename.** The id follows the class name, so renaming stops
+  older plans from decoding. Declare `__datafusion_codec_id__` on the exporting
+  object to pin an id that survives the rename.
+
+`SessionContext.logical_extension_codec_ids()` and its physical counterpart list the
+ids installed on a session, which is also what a decode failure names.
+
+Installing one context's codec stack on another session composes the two sessions
+rather than copying codecs out of one: the imported codecs resolve their task context
+against the original and stop working when it is dropped — see
+[One session, one `Arc<SessionContext>`](#one-session-one-arcsessioncontext). Pass
+the context itself rather than the capsule it exports, so its codecs get an id that
+other sessions can decode.
+
+Because decoding keys off the id rather than install position, registration order
+between independent libraries does not affect decoding at all. It is visible only
+on encoding, where codecs are consulted in install order and the first to claim an
+object wins — so installing a library can claim objects nothing else claimed, but
+never takes over an object an earlier codec was already encoding. Two libraries
+that each own tables, functions, and a planner register like this:
+
+```python
+ctx = SessionContext(config)
+
+# Codecs from both libraries. Order between libraries does not matter.

Review Comment:
   I don't think it's possible because any object decoded by a foreign codec will show up as a `ForeignXYZ` when attempting a downcast so the type ID should never match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3786206600


##########
crates/core/src/codec.rs:
##########
@@ -223,32 +224,129 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Run `f` against each codec in `chain`, returning the first `Ok`.
+///
+/// A codec signals "not mine" by returning an error, so the chain
+/// keeps trying until a codec succeeds. When every codec fails and the
+/// chain has more than one entry, the errors are aggregated into a
+/// single message — returning only the last error would surface the
+/// terminal `Default*ExtensionCodec` "not provided" message and mask
+/// the more specific diagnostic from an installed codec (e.g. a
+/// corrupt-token error from the codec that owns the payload family).
+fn chain_try<C: ?Sized, R>(chain: &[Arc<C>], what: &str, f: impl Fn(&C) -> Result<R>) -> Result<R> {
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {
+        match f(codec) {
+            Ok(value) => return Ok(value),
+            Err(err) => errors.push(err),
+        }
+    }
+    Err(aggregate_chain_errors(what, errors))
+}
+
+/// Collapse per-codec failures into one error. A single failure is
+/// returned as-is so the one-codec (default-only) chain behaves
+/// exactly like the pre-chain implementation.
+fn aggregate_chain_errors(
+    what: &str,
+    mut errors: Vec<datafusion::error::DataFusionError>,
+) -> datafusion::error::DataFusionError {
+    match errors.len() {
+        0 => datafusion::error::DataFusionError::Internal(format!(
+            "Empty extension codec chain while handling {what}"
+        )),
+        1 => errors.swap_remove(0),
+        _ => {
+            let joined = errors
+                .iter()
+                .map(|err| err.to_string())
+                .collect::<Vec<_>>()
+                .join("; ");
+            datafusion::error::DataFusionError::Execution(format!(
+                "None of the {} composed extension codecs handled {what}: {joined}",
+                errors.len()
+            ))
+        }
+    }
+}
+
+/// Encode variant of [`chain_try`] for methods that write into a
+/// caller-provided buffer.
+///
+/// Each codec encodes into a scratch buffer so a failed attempt cannot
+/// leave partial bytes behind. `Ok` with bytes written commits those
+/// bytes and ends the chain. `Ok` with an empty buffer is treated as
+/// "no opinion" — the standard `Default*ExtensionCodec` behavior of
+/// encoding a UDF by name writes nothing — so later codecs still get a
+/// chance to emit a richer payload. If no codec writes bytes but at
+/// least one returned `Ok`, the overall result is `Ok` with nothing
+/// written (encode by name).
+fn chain_encode<C: ?Sized>(
+    chain: &[Arc<C>],
+    buf: &mut Vec<u8>,
+    what: &str,
+    f: impl Fn(&C, &mut Vec<u8>) -> Result<()>,
+) -> Result<()> {
+    let mut saw_empty_ok = false;
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {
+        let mut scratch = Vec::new();
+        match f(codec, &mut scratch) {
+            Ok(()) if !scratch.is_empty() => {
+                buf.extend_from_slice(&scratch);
+                return Ok(());
+            }
+            Ok(()) => saw_empty_ok = true,
+            Err(err) => errors.push(err),
+        }
+    }
+    if saw_empty_ok {
+        return Ok(());
+    }
+    Err(aggregate_chain_errors(what, errors))
+}
+
 /// `LogicalExtensionCodec` parked on every `SessionContext`. Holds
 /// the Python-aware encoding hooks for logical-layer types
 /// (`LogicalPlan`, `Expr`) and delegates everything it does not
-/// handle to the composable `inner` codec — typically
-/// `DefaultLogicalExtensionCodec`, or a downstream FFI codec
-/// installed via `SessionContext.with_logical_extension_codec(...)`.
+/// handle to a chain of composable codecs. The chain starts as just
+/// `DefaultLogicalExtensionCodec`; each downstream FFI codec installed
+/// via `SessionContext.with_logical_extension_codec(...)` is prepended,
+/// so the most recently installed codec is consulted first and the
+/// default codec always runs last.
+///
+/// Chain dispatch relies on each codec recognizing its own payloads
+/// (distinct family prefixes — see the module docs) and returning an
+/// error for everything else so the next codec gets a chance.
 ///
 /// Sitting at the top of the session's logical codec stack means
 /// every serializer that reads `session.logical_codec()` automatically
 /// picks up Python-aware encoding for free.
-#[derive(Debug)]
+#[derive(Debug, Clone)]
 pub struct PythonLogicalCodec {
-    inner: Arc<dyn LogicalExtensionCodec>,
+    chain: Vec<Arc<dyn LogicalExtensionCodec>>,
     python_udf_inlining: bool,
 }
 
 impl PythonLogicalCodec {
     pub fn new(inner: Arc<dyn LogicalExtensionCodec>) -> Self {
         Self {
-            inner,
+            chain: vec![inner],
             python_udf_inlining: true,
         }
     }
 
-    pub fn inner(&self) -> &Arc<dyn LogicalExtensionCodec> {

Review Comment:
   This was pub before. Do people care about inspecting the codecs directly?



##########
docs/source/contributor-guide/ffi.md:
##########
@@ -248,15 +248,65 @@ foreign planner. This lets the planner decode provider-owned objects and lets
 process-local tokens to demonstrate ownership; production codecs should serialize
 durable metadata instead.
 
-The current Python API has one external logical codec and one external physical codec.
-Installing another codec replaces the prior codec rather than composing a registry.
-The example therefore has one external codec owner, and the planner uses built-in
-physical nodes. Install the provider codecs before the planner where possible.
+### Composable codecs
+
+Extension codecs compose. Each call to `with_logical_extension_codec` or
+`with_physical_extension_codec` adds the codec to the front of the session's codec
+chain rather than replacing prior codecs. During encoding and decoding, the most
+recently installed codec is consulted first, falling through codec by codec to
+DataFusion's default codec. A codec signals "not mine" by returning an error, which
+sends the chain on to the next codec. Two conventions keep this dispatch sound:
+
+- Frame your payloads with a distinct byte prefix (pick a `DF` namespace plus a
+  crate-specific suffix) and only decode payloads carrying your prefix.
+- Return an error for objects and payloads you do not own. A codec that answers
+  success for objects outside its family shadows every codec installed before it.
+
+Because dispatch keys off payload prefixes rather than install position, codec
+registration order between independent libraries does not matter.
 
 The current FFI logical codec supports providers and UDFs but not arbitrary custom
 `LogicalPlan::Extension` nodes. See both example READMEs for the supported flow and
 local build commands.
 
+### One planner per session, with explicit fallback
+
+Unlike codecs, a `SessionState` holds exactly one query planner — installing another
+replaces it. Planner layering is therefore explicit: a planner that wants to handle
+only some queries should accept a fallback planner and delegate the rest to it. The
+current planner can be exported for that purpose with
+`ctx.__datafusion_query_planner__()`.
+
+One ordering rule applies: a planner capsule captures the session's codecs at export
+time and cannot be rebound afterward. Codec changes made after installing a single
+planner are rebound automatically, but a planner wrapped inside another planner as a
+fallback is opaque and keeps the codecs it was exported with. **Install all extension
+codecs before exporting or chaining planners.**
+
+Putting it together for a session using two extension libraries that each provide
+tables, functions, and a query planner:
+
+```python
+ctx = SessionContext(config)
+
+# 1. Codecs from both libraries. Order between libraries does not matter.
+ctx = ctx.with_logical_extension_codec(lib_a.codec())
+ctx = ctx.with_logical_extension_codec(lib_b.codec())

Review Comment:
   So is the resulting stage at this point `[default, lib_a, lib_b]`? I could read it or ask claude if some assumed based codec for standard datafusion python gets installed always as a fallback. Mostly I'm curious if it's a reasonable workflow for someone to set things up to only get `[lib_a, lib_b]` and if anything unsupported by their custom codecs barfs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3796279741


##########
crates/core/src/codec.rs:
##########
@@ -223,32 +224,129 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Run `f` against each codec in `chain`, returning the first `Ok`.
+///
+/// A codec signals "not mine" by returning an error, so the chain
+/// keeps trying until a codec succeeds. When every codec fails and the
+/// chain has more than one entry, the errors are aggregated into a
+/// single message — returning only the last error would surface the
+/// terminal `Default*ExtensionCodec` "not provided" message and mask
+/// the more specific diagnostic from an installed codec (e.g. a
+/// corrupt-token error from the codec that owns the payload family).
+fn chain_try<C: ?Sized, R>(chain: &[Arc<C>], what: &str, f: impl Fn(&C) -> Result<R>) -> Result<R> {
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {

Review Comment:
   this is not safe, you can get things decoded by accident.  please dont ask how i know and how much time i spent debuging it 



##########
crates/core/src/codec.rs:
##########
@@ -223,32 +224,129 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Run `f` against each codec in `chain`, returning the first `Ok`.
+///
+/// A codec signals "not mine" by returning an error, so the chain
+/// keeps trying until a codec succeeds. When every codec fails and the
+/// chain has more than one entry, the errors are aggregated into a
+/// single message — returning only the last error would surface the
+/// terminal `Default*ExtensionCodec` "not provided" message and mask
+/// the more specific diagnostic from an installed codec (e.g. a
+/// corrupt-token error from the codec that owns the payload family).
+fn chain_try<C: ?Sized, R>(chain: &[Arc<C>], what: &str, f: impl Fn(&C) -> Result<R>) -> Result<R> {
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {
+        match f(codec) {
+            Ok(value) => return Ok(value),
+            Err(err) => errors.push(err),
+        }
+    }
+    Err(aggregate_chain_errors(what, errors))
+}
+
+/// Collapse per-codec failures into one error. A single failure is
+/// returned as-is so the one-codec (default-only) chain behaves
+/// exactly like the pre-chain implementation.
+fn aggregate_chain_errors(
+    what: &str,
+    mut errors: Vec<datafusion::error::DataFusionError>,
+) -> datafusion::error::DataFusionError {
+    match errors.len() {
+        0 => datafusion::error::DataFusionError::Internal(format!(
+            "Empty extension codec chain while handling {what}"
+        )),
+        1 => errors.swap_remove(0),
+        _ => {
+            let joined = errors
+                .iter()
+                .map(|err| err.to_string())
+                .collect::<Vec<_>>()
+                .join("; ");
+            datafusion::error::DataFusionError::Execution(format!(
+                "None of the {} composed extension codecs handled {what}: {joined}",
+                errors.len()
+            ))
+        }
+    }
+}
+
+/// Encode variant of [`chain_try`] for methods that write into a
+/// caller-provided buffer.
+///
+/// Each codec encodes into a scratch buffer so a failed attempt cannot
+/// leave partial bytes behind. `Ok` with bytes written commits those
+/// bytes and ends the chain. `Ok` with an empty buffer is treated as
+/// "no opinion" — the standard `Default*ExtensionCodec` behavior of
+/// encoding a UDF by name writes nothing — so later codecs still get a
+/// chance to emit a richer payload. If no codec writes bytes but at
+/// least one returned `Ok`, the overall result is `Ok` with nothing
+/// written (encode by name).
+fn chain_encode<C: ?Sized>(

Review Comment:
   this might work, but still can be tricky



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#issuecomment-5499084674

   @milenkovicm @gabotechs @ntjohnson1 I think I've addressed the comments, and I think this is a much better setup now. Thanks for the push back!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#issuecomment-5543795280

   Ok I think we're good with all the comments, some deferred to follow up issues. Thank you for the reviews @ntjohnson1 and @milenkovicm !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3918593844


##########
examples/datafusion-ffi-example/python/tests/_test_logical_extension_codec.py:
##########
@@ -80,3 +118,430 @@ def test_ffi_logical_codec_roundtrip():
     restored = LogicalPlan.from_bytes(ctx, blob)
     df_round_trip = ctx.create_dataframe_from_logical_plan(restored)
     assert df.collect() == df_round_trip.collect()
+
+
+def test_ffi_logical_codec_composes_with_later_install():
+    """Codecs compose: installing a second codec appends it to the
+    session's codec chain instead of replacing the first. The second
+    codec here (a default-backed codec exported from a fresh session)
+    cannot encode this library's table provider, so the first codec
+    still claims it. Under replace semantics this test fails with
+    `LogicalExtensionCodec is not provided`."""
+    ctx, codec = _setup_session_with_codec()
+    ctx = ctx.with_logical_extension_codec(
+        SessionContext().__datafusion_logical_extension_codec__()
+    )
+
+    ctx.register_table("numbers", MyTableProvider(1, 4, 1))
+    df = ctx.sql('SELECT "A" FROM numbers')
+    plan = df.logical_plan()
+
+    before = codec.table_provider_encode_calls()
+    blob = plan.to_bytes(ctx)
+    assert codec.table_provider_encode_calls() > before
+
+    restored = LogicalPlan.from_bytes(ctx, blob)
+    df_round_trip = ctx.create_dataframe_from_logical_plan(restored)
+    assert df.collect() == df_round_trip.collect()

Review Comment:
   nit, should you assert on decode call number as well?



##########
examples/datafusion-ffi-example/src/name_only_codec.rs:
##########
@@ -0,0 +1,268 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A codec whose functions need no payload at all.
+//!
+//! Most extension codecs answer with bytes. This one owns a fixed catalog of
+//! functions that are fully described by their names, so `try_encode_udf`
+//! writes nothing and `try_decode_udf` rebuilds the function from `name`
+//! alone. DataFusion supports that shape directly: an encoder that writes no
+//! bytes leaves `fun_definition` unset, and the decoder then tries the
+//! `FunctionRegistry` first and the codec second — see the
+//! `None => ctx.udf(..).or_else(|_| codec.try_decode_udf(name, &[]))` arm in
+//! `datafusion-proto`'s `from_proto.rs`.
+//!
+//! It exists here to pin that arm. Because there are no bytes, there is
+//! nothing to tag with the codec's identity, so this is the one path where
+//! `PythonLogicalCodec` still offers a payload to every installed codec in
+//! turn. A change that wrapped empty encodings in an envelope would set
+//! `fun_definition`, skip the registry lookup permanently, and break both this
+//! codec and plain by-name round trips — with no other test noticing.
+
+use std::fmt;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+use arrow_schema::DataType;
+use datafusion::common::error::Result;
+use datafusion::common::not_impl_err;
+use datafusion::logical_expr::{
+    ColumnarValue, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, TypeSignature,
+    Volatility,
+};
+use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec;
+use datafusion_proto::logical_plan::{DefaultLogicalExtensionCodec, LogicalExtensionCodec};
+use datafusion_python_util::{ffi_task_context_provider_from_pycapsule, get_tokio_runtime};
+use pyo3::prelude::*;
+use pyo3::types::PyCapsule;
+
+/// Prefix marking the functions this library owns. A name is the entire
+/// encoding, so the prefix is the whole ownership test.
+const NAME_PREFIX: &str = "name_only_";
+
+/// Scalar function reconstructed purely from its name.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+struct NameOnlyUdf {
+    name: String,
+    signature: Signature,
+}
+
+impl NameOnlyUdf {
+    fn new(name: impl Into<String>) -> Self {
+        Self {
+            name: name.into(),
+            signature: Signature::new(TypeSignature::Any(1), Volatility::Immutable),
+        }
+    }
+}
+
+impl ScalarUDFImpl for NameOnlyUdf {
+    fn name(&self) -> &str {
+        &self.name
+    }
+
+    fn signature(&self) -> &Signature {
+        &self.signature
+    }
+
+    fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+        Ok(DataType::Int64)
+    }
+
+    fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
+        Ok(args.args[0].clone())
+    }
+}
+
+#[derive(Default)]
+struct Counters {
+    encode_udf: AtomicUsize,
+    decode_udf: AtomicUsize,
+}
+
+impl fmt::Debug for Counters {
+    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
+        formatter.debug_struct("Counters").finish_non_exhaustive()
+    }
+}
+
+struct NameOnlyLogicalExtensionCodec {
+    inner: DefaultLogicalExtensionCodec,
+    counters: Arc<Counters>,
+}
+
+impl fmt::Debug for NameOnlyLogicalExtensionCodec {
+    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
+        formatter
+            .debug_struct("NameOnlyLogicalExtensionCodec")
+            .finish_non_exhaustive()
+    }
+}
+
+impl LogicalExtensionCodec for NameOnlyLogicalExtensionCodec {
+    fn try_decode(
+        &self,
+        buf: &[u8],
+        inputs: &[datafusion::logical_expr::LogicalPlan],
+        ctx: &datafusion::execution::TaskContext,
+    ) -> Result<datafusion::logical_expr::Extension> {
+        self.inner.try_decode(buf, inputs, ctx)
+    }
+
+    fn try_encode(
+        &self,
+        node: &datafusion::logical_expr::Extension,
+        buf: &mut Vec<u8>,
+    ) -> Result<()> {
+        self.inner.try_encode(node, buf)
+    }
+
+    fn try_decode_table_provider(
+        &self,
+        buf: &[u8],
+        table_ref: &datafusion::common::TableReference,
+        schema: arrow_schema::SchemaRef,
+        ctx: &datafusion::execution::TaskContext,
+    ) -> Result<Arc<dyn datafusion::datasource::TableProvider>> {
+        self.inner
+            .try_decode_table_provider(buf, table_ref, schema, ctx)
+    }
+
+    fn try_encode_table_provider(
+        &self,
+        table_ref: &datafusion::common::TableReference,
+        node: Arc<dyn datafusion::datasource::TableProvider>,
+        buf: &mut Vec<u8>,
+    ) -> Result<()> {
+        self.inner.try_encode_table_provider(table_ref, node, buf)
+    }
+
+    /// Writes nothing on purpose. The name is the whole encoding, so there is
+    /// no payload to emit, and returning `Ok` with an empty buffer is how a
+    /// codec says "encoded by name" to DataFusion.
+    fn try_encode_udf(&self, node: &ScalarUDF, _buf: &mut Vec<u8>) -> Result<()> {
+        if node.name().starts_with(NAME_PREFIX) {
+            self.counters.encode_udf.fetch_add(1, Ordering::SeqCst);
+        }
+        Ok(())
+    }
+
+    /// Rebuilds the function from `name`, with no registry entry and no bytes.
+    fn try_decode_udf(&self, name: &str, buf: &[u8]) -> Result<Arc<ScalarUDF>> {
+        if !name.starts_with(NAME_PREFIX) {
+            return not_impl_err!("Not a name-only function: {name}");
+        }
+        if !buf.is_empty() {
+            return not_impl_err!(
+                "name-only functions carry no payload, but {} bytes were supplied for {name}",
+                buf.len()
+            );
+        }
+        self.counters.decode_udf.fetch_add(1, Ordering::SeqCst);
+        Ok(Arc::new(ScalarUDF::from(NameOnlyUdf::new(name))))
+    }
+}
+
+/// The function [`NameOnlyUdfCodec`] owns, exported so a session can register
+/// it and build a plan that references it.
+///
+/// Only the *encoding* session needs it registered. The decoding session
+/// deliberately does not, which is what forces the codec's name-only decode
+/// path to run.
+#[pyclass(
+    from_py_object,
+    name = "NameOnlyFunction",
+    module = "datafusion_ffi_example",
+    subclass
+)]
+#[derive(Debug, Clone)]
+pub(crate) struct NameOnlyFunction;
+
+#[pymethods]
+impl NameOnlyFunction {
+    #[new]
+    fn new() -> Self {
+        Self
+    }
+
+    fn __datafusion_scalar_udf__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyCapsule>> {
+        let func = Arc::new(ScalarUDF::from(NameOnlyUdf::new(format!(
+            "{NAME_PREFIX}identity"
+        ))));
+        PyCapsule::new_with_value(
+            py,
+            datafusion_ffi::udf::FFI_ScalarUDF::from(func),
+            cr"datafusion_scalar_udf",
+        )
+    }
+}
+
+/// Codec owning functions that are reconstructible from their names alone.
+///
+/// A real library shaped like this would be one shipping a fixed catalog of
+/// built-ins: nothing about a call site varies, so there is nothing to encode.
+#[pyclass(
+    from_py_object,
+    name = "NameOnlyUdfCodec",
+    module = "datafusion_ffi_example",
+    subclass
+)]
+#[derive(Clone)]
+pub(crate) struct NameOnlyUdfCodec {
+    counters: Arc<Counters>,
+}
+
+#[pymethods]
+impl NameOnlyUdfCodec {
+    #[new]
+    fn new() -> Self {
+        Self {
+            counters: Arc::new(Counters::default()),
+        }
+    }
+
+    /// Name of the function this codec can rebuild, for use in a query.
+    #[staticmethod]
+    fn function_name() -> String {
+        format!("{NAME_PREFIX}identity")
+    }
+
+    fn encode_udf_calls(&self) -> usize {
+        self.counters.encode_udf.load(Ordering::SeqCst)
+    }
+
+    fn decode_udf_calls(&self) -> usize {
+        self.counters.decode_udf.load(Ordering::SeqCst)
+    }
+
+    fn __datafusion_logical_extension_codec__<'py>(

Review Comment:
   thanks for example



##########
examples/datafusion-ffi-query-planner-example/python/tests/_test_three_library_query_planner.py:
##########
@@ -667,3 +686,26 @@ def test_query_planner_rejects_invalid_config(max_rows: str):
 
     with pytest.raises(Exception, match=r"max_rows|Invalid value"):
         ctx.sql(f"SET ffi_query_planner.max_rows = '{max_rows}'").collect()
+
+
+def test_composed_codecs_with_query_planner():
+    """A second pair of codecs installed on top of the provider codecs
+    composes with them instead of replacing them. The extra codecs
+    (default-backed exports from a fresh session) decline everything,
+    so planner-driven encode/decode falls through to the provider

Review Comment:
   decline everything but should be called? Am I correct?



##########
crates/core/src/codec.rs:
##########
@@ -223,12 +233,339 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Family prefix for the envelope wrapping a chained codec's payload.
+///
+/// A distinct magic is what makes "is this framed?" a definite test
+/// rather than a speculative decode. Probing by attempting to parse the
+/// envelope would reintroduce exactly the protobuf ambiguity this
+/// framing exists to remove: prost skips unknown fields and defaults
+/// missing ones, so a foreign payload can parse cleanly as an envelope.
+pub(crate) const CHAINED_PAYLOAD_FAMILY: &[u8] = b"DFPYCHN";
+
+/// Wire-format version for the chained-payload envelope. Independent of
+/// [`WIRE_VERSION_CURRENT`], which versions the cloudpickle framing.
+pub(crate) const CHAIN_WIRE_VERSION_CURRENT: u8 = 1;
+
+/// Oldest chained-payload envelope version this build decodes.
+pub(crate) const CHAIN_WIRE_VERSION_MIN_SUPPORTED: u8 = 1;
+
+/// Prefix for the synthetic id given to a codec installed from a bare
+/// PyCapsule, which exposes nothing stable to derive an identity from.
+/// The rest of the id is random per install, so no other session can
+/// mint it: a payload carrying one decodes within the installing
+/// session's lineage, which clones the id along with the chain, and
+/// fails with a pointed error anywhere else rather than resolving to a
+/// different codec. A counter or a chain position would not do — every
+/// session numbers from the same end, so the first bare capsule
+/// installed anywhere would answer for every other session's first.

Review Comment:
   its hard for me to make sense of this comment, it has quite a "legal tone" for me (non english native)



##########
crates/core/src/codec.rs:
##########
@@ -321,30 +724,54 @@ impl LogicalExtensionCodec for PythonLogicalCodec {
         node: Arc<dyn TableProvider>,
         buf: &mut Vec<u8>,
     ) -> Result<()> {
-        self.inner.try_encode_table_provider(table_ref, node, buf)
+        chain_encode(
+            &self.chain,
+            &self.terminal,
+            buf,
+            "a table provider",
+            |codec, buf| codec.try_encode_table_provider(table_ref, Arc::clone(&node), buf),
+        )
     }
 
     fn try_decode_file_format(
         &self,
         buf: &[u8],
         ctx: &TaskContext,
     ) -> Result<Arc<dyn FileFormatFactory>> {
-        self.inner.try_decode_file_format(buf, ctx)
+        chain_decode(
+            &self.chain,
+            &self.terminal,
+            buf,
+            "a file format",
+            |codec, buf| codec.try_decode_file_format(buf, ctx),
+        )
     }
 
     fn try_encode_file_format(
         &self,
         buf: &mut Vec<u8>,
         node: Arc<dyn FileFormatFactory>,
     ) -> Result<()> {
-        self.inner.try_encode_file_format(buf, node)
+        chain_encode(
+            &self.chain,
+            &self.terminal,
+            buf,
+            "a file format",
+            |codec, buf| codec.try_encode_file_format(buf, Arc::clone(&node)),
+        )
     }
 
     fn try_encode_udf(&self, node: &ScalarUDF, buf: &mut Vec<u8>) -> Result<()> {
         if self.python_udf_inlining && try_encode_python_scalar_udf(node, buf)? {

Review Comment:
   i believe python udf inlining could be extracted to a separate codec, and then codec could be added to chain of codecs, when inlining is enabled. 
   
   no need to change it now, just thinking aloud



##########
docs/source/contributor-guide/ffi.md:
##########
@@ -248,10 +248,96 @@ foreign planner. This lets the planner decode provider-owned objects and lets
 process-local tokens to demonstrate ownership; production codecs should serialize
 durable metadata instead.
 
-The current Python API has one external logical codec and one external physical codec.
-Installing another codec replaces the prior codec rather than composing a registry.
-The example therefore has one external codec owner, and the planner uses built-in
-physical nodes. Install the provider codecs before the planner where possible.
+### Composable codecs
+
+Extension codecs compose. Each call to `with_logical_extension_codec` or
+`with_physical_extension_codec` appends the codec to the session's codec chain
+rather than replacing prior codecs.
+
+**Nothing is asked of the codec itself.** Implement `LogicalExtensionCodec` or
+`PhysicalExtensionCodec` exactly as you would for a session that installs only
+yours. When your codec writes bytes into a serialized plan, datafusion-python
+records which codec wrote them, and strips that record off again before handing the
+bytes back. So your codec receives, byte for byte, the payload it wrote, and is
+never offered a payload another codec wrote.
+
+A codec that also ships to hosts which dispatch differently may still want its own
+guard against foreign payloads. Keeping one is fine; it is simply not needed for the
+datafusion-python path.
+
+That record is the codec's **id**: a short string stored inside the plan, naming the
+codec that wrote each payload. Because plans are decoded in another process — or
+another program — the id has to name the same codec there as it did where the plan
+was written.
+
+Ids are assigned for you. A codec's id is normally its exporting class's import
+path, such as `my_library.Codec`, which is what you will see in
+`logical_extension_codec_ids()` and in decode errors. You choose one yourself in
+three cases:
+
+- **Two instances of one class.** Both get the same id, so the second install
+  raises `ValueError`. Pass `codec_id=` to tell them apart.
+- **A bare `PyCapsule`.** A capsule has no class to take a name from, so it gets an
+  id private to the session that installed it. Plans it encodes fail with a clear
+  error on any other session, rather than being decoded by the wrong codec. Pass
+  `codec_id=` if those plans have to cross sessions.
+- **A class you intend to rename.** The id follows the class name, so renaming stops
+  older plans from decoding. Declare `__datafusion_codec_id__` on the exporting
+  object to pin an id that survives the rename.
+
+`SessionContext.logical_extension_codec_ids()` and its physical counterpart list the
+ids installed on a session, which is also what a decode failure names.
+
+Installing one context's codec stack on another session composes the two sessions
+rather than copying codecs out of one: the imported codecs resolve their task context
+against the original and stop working when it is dropped — see
+[One session, one `Arc<SessionContext>`](#one-session-one-arcsessioncontext). Pass
+the context itself rather than the capsule it exports, so its codecs get an id that
+other sessions can decode.
+
+Because decoding keys off the id rather than install position, registration order
+between independent libraries does not affect decoding at all. It is visible only
+on encoding, where codecs are consulted in install order and the first to claim an
+object wins — so installing a library can claim objects nothing else claimed, but
+never takes over an object an earlier codec was already encoding. Two libraries
+that each own tables, functions, and a planner register like this:
+
+```python
+ctx = SessionContext(config)
+
+# Codecs from both libraries. Order between libraries does not matter.

Review Comment:
   Most of the codecs do lot of `downcast_ref` which under to hood compares `TypeId` before it tries to access structure properties. With chained encoders, Is there a chance that we have `TypeId` collision,  with two different libraries have a different structures with same `TypeId`, triggering encoding in the wrong encoder? 
   
   I guess even its possible it is low probability, this is a bit of philosophical question 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3905881375


##########
crates/core/src/codec.rs:
##########
@@ -223,32 +224,129 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Run `f` against each codec in `chain`, returning the first `Ok`.
+///
+/// A codec signals "not mine" by returning an error, so the chain
+/// keeps trying until a codec succeeds. When every codec fails and the
+/// chain has more than one entry, the errors are aggregated into a
+/// single message — returning only the last error would surface the
+/// terminal `Default*ExtensionCodec` "not provided" message and mask
+/// the more specific diagnostic from an installed codec (e.g. a
+/// corrupt-token error from the codec that owns the payload family).
+fn chain_try<C: ?Sized, R>(chain: &[Arc<C>], what: &str, f: impl Fn(&C) -> Result<R>) -> Result<R> {
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {

Review Comment:
   You were both right, and I've reworked the dispatch rather than keeping trial decoding. Thanks for the #16980 / #16986 pointers — the CSV-decoded-as-Parquet case is exactly the failure mode, and it's not one a codec can defend against from inside, since `MyMessage::decode(buf)` has no prefix to check and cannot decline.
   
   Every payload an installed codec writes is now wrapped in an envelope naming its author: `DFPYCHN | version: u8 | id_len: u32 (LE) | id | blob`. Decoding reads the id and consults exactly one codec. Nothing is ever offered to a codec that did not write it, so structurally similar prost messages can't cross libraries. Codecs themselves are unchanged — they receive their payload byte for byte and never see the envelope.
   
   On reusing `ComposedPhysicalExtensionCodec`: I looked at it closely and it doesn't fit here, for three reasons.
   
   1. It keys on `encoder_position`, which is sound when both ends build the same list in the same order. That's true for the consumers it was written for — Ballista's list is a compile-time constant, and datafusion-distributed pins its own codec at index 0 and appends user codecs rebuilt from the same startup code on every node. It isn't true here: the chain is assembled by user Python, and `Expr.to_bytes(ctx1)` / `Expr.from_bytes(ctx2)` puts two independently configured sessions on either end of one payload. An index names a different codec in the decoder as soon as install order differs. `test_decode_survives_a_different_install_order` is the case.
   2. It only exists for the physical layer. There's no `ComposedLogicalExtensionCodec`, and this needs both — table providers and UDFs go through the logical codec.
   3. `encode_protobuf` always wraps in `DataEncoderTuple`, so a codec that writes zero bytes still emits a non-empty payload. That sets `fun_definition` and permanently skips the `FunctionRegistry` lookup the decoder does first, which breaks DataFusion's encode-by-name path. I need that path to keep working — `NameOnlyUdfCodec` in the FFI example owns functions that are fully described by their names and encodes nothing at all.
   
   The reasoning is recorded in the code at `crates/core/src/codec.rs` (see `ChainEntry` and `chain_decode`, which cite both upstream issues) so the next person doesn't re-derive it, and the "Composable codecs" section of `docs/source/contributor-guide/ffi.md` covers it for extension authors. The empty-payload case is the one place codecs are still consulted in turn, because there are no bytes to carry an identity; that question is "do you own the function named `x`", which is name-scoped, and two codecs disagreeing means they've already collided in the function registry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3894481979


##########
crates/core/src/codec.rs:
##########
@@ -223,32 +224,129 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Run `f` against each codec in `chain`, returning the first `Ok`.
+///
+/// A codec signals "not mine" by returning an error, so the chain
+/// keeps trying until a codec succeeds. When every codec fails and the
+/// chain has more than one entry, the errors are aggregated into a
+/// single message — returning only the last error would surface the
+/// terminal `Default*ExtensionCodec` "not provided" message and mask
+/// the more specific diagnostic from an installed codec (e.g. a
+/// corrupt-token error from the codec that owns the payload family).
+fn chain_try<C: ?Sized, R>(chain: &[Arc<C>], what: &str, f: impl Fn(&C) -> Result<R>) -> Result<R> {
+    let mut errors: Vec<datafusion::error::DataFusionError> = Vec::new();
+    for codec in chain {

Review Comment:
   ☝️ this is important, some more context in https://github.com/apache/datafusion/issues/16980 and https://github.com/apache/datafusion/pull/16986.
   
   Because of how protobuf decoding works, it's very easy to be in situations where the same protobuf payload can decode to multiple `prost` structs, specially in cases where the structs are specially simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3936188655


##########
crates/core/src/codec.rs:
##########
@@ -321,30 +724,54 @@ impl LogicalExtensionCodec for PythonLogicalCodec {
         node: Arc<dyn TableProvider>,
         buf: &mut Vec<u8>,
     ) -> Result<()> {
-        self.inner.try_encode_table_provider(table_ref, node, buf)
+        chain_encode(
+            &self.chain,
+            &self.terminal,
+            buf,
+            "a table provider",
+            |codec, buf| codec.try_encode_table_provider(table_ref, Arc::clone(&node), buf),
+        )
     }
 
     fn try_decode_file_format(
         &self,
         buf: &[u8],
         ctx: &TaskContext,
     ) -> Result<Arc<dyn FileFormatFactory>> {
-        self.inner.try_decode_file_format(buf, ctx)
+        chain_decode(
+            &self.chain,
+            &self.terminal,
+            buf,
+            "a file format",
+            |codec, buf| codec.try_decode_file_format(buf, ctx),
+        )
     }
 
     fn try_encode_file_format(
         &self,
         buf: &mut Vec<u8>,
         node: Arc<dyn FileFormatFactory>,
     ) -> Result<()> {
-        self.inner.try_encode_file_format(buf, node)
+        chain_encode(
+            &self.chain,
+            &self.terminal,
+            buf,
+            "a file format",
+            |codec, buf| codec.try_encode_file_format(buf, Arc::clone(&node)),
+        )
     }
 
     fn try_encode_udf(&self, node: &ScalarUDF, buf: &mut Vec<u8>) -> Result<()> {
         if self.python_udf_inlining && try_encode_python_scalar_udf(node, buf)? {

Review Comment:
   I opened a follow up issue. great idea!



##########
crates/core/src/codec.rs:
##########
@@ -223,12 +233,339 @@ fn strip_wire_header<'a>(
     Ok(Some(&buf[py_minor_idx + 1..]))
 }
 
+/// Family prefix for the envelope wrapping a chained codec's payload.
+///
+/// A distinct magic is what makes "is this framed?" a definite test
+/// rather than a speculative decode. Probing by attempting to parse the
+/// envelope would reintroduce exactly the protobuf ambiguity this
+/// framing exists to remove: prost skips unknown fields and defaults
+/// missing ones, so a foreign payload can parse cleanly as an envelope.
+pub(crate) const CHAINED_PAYLOAD_FAMILY: &[u8] = b"DFPYCHN";
+
+/// Wire-format version for the chained-payload envelope. Independent of
+/// [`WIRE_VERSION_CURRENT`], which versions the cloudpickle framing.
+pub(crate) const CHAIN_WIRE_VERSION_CURRENT: u8 = 1;
+
+/// Oldest chained-payload envelope version this build decodes.
+pub(crate) const CHAIN_WIRE_VERSION_MIN_SUPPORTED: u8 = 1;
+
+/// Prefix for the synthetic id given to a codec installed from a bare
+/// PyCapsule, which exposes nothing stable to derive an identity from.
+/// The rest of the id is random per install, so no other session can
+/// mint it: a payload carrying one decodes within the installing
+/// session's lineage, which clones the id along with the chain, and
+/// fails with a pointed error anywhere else rather than resolving to a
+/// different codec. A counter or a chain position would not do — every
+/// session numbers from the same end, so the first bare capsule
+/// installed anywhere would answer for every other session's first.

Review Comment:
   Tried to clean it up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Make extension codecs composable [datafusion-python]

Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on code in PR #1678:
URL: https://github.com/apache/datafusion-python/pull/1678#discussion_r3786267798


##########
examples/datafusion-ffi-example/README.md:
##########
@@ -35,7 +35,9 @@ Separate shared libraries guarantee distinct DataFusion library markers. This ca
 
 The example codecs do not inspect the callback `TaskContext`. A production codec that depends on session configuration or registered functions must ensure its exported FFI codec is bound to, and retains, the appropriate host `TaskContextProvider`.
 
-The current Python API installs one external logical codec and one external physical codec. It does not yet compose codecs from several independent plugin owners. This example therefore makes the provider library the sole external codec owner; the planner uses built-in physical nodes and receives the provider codecs from the host.
+Extension codecs compose: each `with_logical_extension_codec` / `with_physical_extension_codec` call prepends the codec to the session's codec chain, with the most recently installed codec consulted first and DataFusion's default codec as the terminal fallback. A codec signals "not mine" by returning an error, so several independent plugin libraries can install codecs on the same session as long as each only answers for payloads it owns (frame them with a distinct byte prefix). In this example the provider library is the only codec owner; the planner uses built-in physical nodes and receives the provider codecs from the host.

Review Comment:
   Ok this line says DF default codec is the terminal fallback but I think the constructor takes a codec in rust at least so it could potentially be updated there but maybe not in the exposed python surface.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org