You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/04/24 20:14:41 UTC

[I] Split proto serialization to encapsulate private state [datafusion]

adriangb opened a new issue, #21835:
URL: https://github.com/apache/datafusion/issues/21835

   # Proposal: split `datafusion-proto` and move serialization onto `PhysicalExpr`
   
   ## Problem
   
   `datafusion-proto` serializes every built-in `PhysicalExpr` through a single ~300-line `downcast_ref` chain in [`serialize_physical_expr_with_converter`](https://github.com/apache/datafusion/blob/main/datafusion/proto/src/physical_plan/to_proto.rs#L255), with a symmetric `match` on `ExprType` in `from_proto.rs`. Because the serializer lives outside the crate where each expression is defined, **every piece of internal state an expression wants to round-trip has to be `pub`**.
   
   The concrete incident that motivated this issue was https://github.com/apache/datafusion/pull/21807, where supporting proto round-tripping of `DynamicFilterPhysicalExpr` required exposing `pub struct Inner`, `pub fn inner()`, `pub fn from_parts()`, `pub fn original_children()`, `pub fn remapped_children()` — all marked "warning: not stable; proto-only" in the docstrings. See discussion at https://github.com/apache/datafusion/pull/21807#discussion_r3138256061 / https://github.com/apache/datafusion/pull/21807#discussion_r3139926953.
   
   The underlying issue is not specific to `DynamicFilterPhysicalExpr`. Any stateful expression (one whose round-trip state isn't expressible through its normal public constructor) will hit the same wall. It also means:
   
   - Built-in expressions use a big central switch; third-party expressions go through `PhysicalExtensionCodec::try_encode_expr`. Two different shapes for the same thing.
   - Every new built-in `PhysicalExpr` requires editing `datafusion-proto`, which many contributors don't touch for their "real" change.
   - The `.proto` schema for each expression lives far away from the expression itself.
   
   ## Proposal
   
   Two changes, in sequence.
   
   ### 1. Extract `datafusion-proto-models`
   
   Mirror the existing `datafusion-proto-common` split, but for the physical/logical plan schemas. The new crate contains only:
   
   - the `.proto` file(s) and the `prost`-generated Rust types,
   - optional `pbjson`/`serde` derives behind a `json` feature,
   - zero datafusion deps beyond `datafusion-proto-common`.
   
   `datafusion-proto` keeps its current public API by re-exporting from `datafusion-proto-models`. Downstream consumers (`datafusion-ffi`, `datafusion-examples`, `benchmarks`) need no changes.
   
   This is a pure refactor — no semantic change, no behavior change.
   
   ### 2. Add `PhysicalExpr::to_proto` (feature-gated)
   
   Add a method to the `PhysicalExpr` trait, gated on a new `proto` feature:
   
   ```rust
   #[cfg(feature = "proto")]
   fn to_proto(
       &self,
       ctx: &dyn PhysicalExprEncoder,
   ) -> Result<Option<PhysicalExprNode>> {
       Ok(None)
   }
   ```
   
   - Default returns `Ok(None)` → "fall through to the existing codec path" (matches today's behavior for extension expressions).
   - Explicit `Ok(Some(...))` → the expression has serialized itself.
   - Expressions that should serialize but don't implement this override the default with `internal_err!("{typename} does not implement to_proto")`.
   
   `PhysicalExprEncoder` is a small trait defined in `datafusion-proto-models`, wrapping the existing `PhysicalExtensionCodec` + `PhysicalProtoConverterExtension` plumbing, with helpers for `encode_child`, `encode_udf`, `encode_udaf`, `encode_udwf`. This keeps `physical-expr-common` free of `datafusion-proto` as a dep.
   
   The existing `serialize_physical_expr_with_converter` tries `expr.to_proto(ctx)` first; if it returns `Ok(None)`, it falls through to the current downcast chain. This lets expressions migrate one at a time without breaking anything.
   
   ### What this unlocks
   
   - **Private state stays private.** `DynamicFilterPhysicalExpr::to_proto` reads the `RwLock` directly — no more `pub struct Inner`. Once migrated, #21807's "pub for proto" scaffolding (`Inner`, `from_parts`, `inner()`, `original_children`, `remapped_children`) can be reverted.
   - **One-file changes.** Adding serialization for a new expression is a method impl next to the expression, not a round trip through `datafusion-proto`.
   - **Parity between built-in and third-party.** Everyone uses the same hook shape.
   
   ## Decode side (follow-up)
   
   The encode-side win is clear. The decode side is still a central `match` on `ExprType`, which is fine — `oneof ExprType` gives us an exhaustive Rust enum, a strict improvement over today's runtime downcast chain.
   
   As a follow-up (probably best after the encode migrations land), we can push decoding back to the expressions too, via associated fns like:
   
   ```rust
   impl BinaryExpr {
       fn from_proto(
           proto: &PhysicalBinaryExprNode,
           ctx: &ProtoDecodeCtx<'_>,
       ) -> Result<Arc<Self>> { ... }
   }
   ```
   
   The central match becomes a dispatch table. The main payoff is that **public god-constructors for private state go away**: `DynamicFilterPhysicalExpr::from_proto` reads the proto fields and builds `Inner` internally, so we can fully delete the `from_parts`/`Inner`-as-pub scaffolding.
   
   ## Alternatives considered
   
   1. **Bytes-level `try_encode_self` hook, no `prost` in `physical-expr`.** Each expression encodes into an opaque `Vec<u8>`. Preserves privacy without pulling `prost` into more crates. Downside: you lose the single `.proto` file as the schema source of truth — cross-language debuggability and interop suffer. Rejected.
   2. **Keep the central switch, mark unstable accessors `#[doc(hidden)]`.** Cheap. Doesn't fix the third-party/built-in asymmetry or the central-edit-per-new-expression problem. Reasonable as a stopgap, not a solution.
   3. **Separate `PhysicalExprProto` trait with a blanket fallback impl.** Would avoid adding to `PhysicalExpr`. Requires a runtime "does it implement this" check (narrow `Any` cast). Slightly more complex than option (2) above and delivers the same thing.
   
   ## Scope
   
   This issue proposes the change only for `PhysicalExpr`. `ExecutionPlan`, `LogicalPlan`, and logical `Expr` have the same shape and would benefit from the same treatment, but each is its own migration (~40 `ExecutionPlan` impls alone). Worth tackling after the PhysicalExpr path is proven.
   
   `datafusion-substrait` is unaffected — it operates independently and does not share code with `datafusion-proto`.
   
   ## Open questions
   
   - Should `PhysicalExprEncoder` live in `datafusion-proto-models` or be a narrow sub-trait that `PhysicalExtensionCodec` re-implements? Leaning toward the former for minimum dep surface.
   - Window / aggregate expressions are serialized through a separate code path in `to_proto.rs` and are not regular `PhysicalExpr` impls. Probably get a parallel `WindowExpr::to_proto` hook, or stay central. Defer until the core `PhysicalExpr` migration is done.
   - `proto` feature default. I'd default it off on `physical-expr-common`/`physical-expr`/`physical-plan` (so developers who don't care pay nothing) and flip it on from `datafusion-proto`. CI would test with the feature on; off-mode is a "it still compiles" smoke test.
   
   ## Related
   
   - #17713 (remove `datafusion` dependency from `datafusion-proto`) — this refactor is consistent with that direction.
   - #21807 (the PR that motivated this discussion).
   - #20418 (serialize+dedupe dynamic filters — the underlying feature work #21807 addresses).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21835:
URL: https://github.com/apache/datafusion/issues/21835#issuecomment-4316159055

   @timsaucer I'm curious your thoughts


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21835:
URL: https://github.com/apache/datafusion/issues/21835#issuecomment-4356856194

   I spoke with Andrew about this.
   He suggested that maybe instead of implementing `to_proto`/`from_proto` we could implement serde traits.
   This is also a big change but it might have the upshot of decoupling from protobuf, which could both simplify our crate dependency graph and enable better support for json, etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "jayshrivastava (via GitHub)" <gi...@apache.org>.
jayshrivastava commented on issue #21835:
URL: https://github.com/apache/datafusion/issues/21835#issuecomment-4356562997

   Overall, I support having `from_proto` and `to_proto` to avoid leaking internals. 
   
   Merging `datafusion-proto-common` into `datafusion-proto`/`datafusion-proto-models` makes sense too. I doubt there's many use cases for importing `datafusion-proto-common` to serialize primitives without importing `datafusion-proto` for plan nodes / expressions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21835:
URL: https://github.com/apache/datafusion/issues/21835#issuecomment-4348350257

   ## Follow-up: Consolidate proto types and remove `FromProto`/`TryFromProto`
   
   In #21929 the proto-models extraction ran into the orphan rule on a number of `impl From<&protobuf::X> for Y` blocks where both sides became foreign to `datafusion-proto`. We worked around this with a pair of local `FromProto<T>` / `TryFromProto<T>` traits in a new `datafusion_proto::convert` module, mirroring `From` / `TryFrom`. That gets the PR to compile and preserves wire compatibility, but it's a workaround rather than a clean architecture: callers see `Y::from_proto(p)` instead of `Y::from(p)`, and the impls live in `datafusion-proto` rather than next to the target types they convert into.
   
   This issue tracks the follow-up cleanup that gets us back to standard `From`/`TryFrom` everywhere, with each impl living next to either its target type or its proto type.
   
   ### What blocks doing it in #21929
   
   The blocker is dep-graph cycles, not the orphan rule by itself. Each option that keeps standard `From`/`TryFrom` syntax requires *somewhere* the impl can live where one of the types is local. The natural homes (the target-type crates with a `proto` feature) cycle today:
   
   ```
   datafusion-proto-models   ──depends on──▶   datafusion-proto-common   ──depends on──▶   datafusion-common
          ▲                                                                                       │
          └────────────────────── if datafusion-common[proto] depends on it ─────────────────────┘
                                             (cycle)
   ```
   
   So `datafusion-common[proto] → datafusion-proto-models` is rejected by cargo because `datafusion-proto-models → datafusion-proto-common → datafusion-common` already exists in the other direction.
   
   This is the same cycle re-emerging anywhere we'd want a target-type crate to host From/TryFrom impls touching `datafusion-common` types.
   
   ### The clean architecture
   
   Collapse `datafusion-proto-common` into `datafusion-proto-models` and push impls down to target-type crates:
   
   | Crate | Owns | Deps |
   |---|---|---|
   | `datafusion-proto-models` | All prost-generated types from both `.proto` files. **Arrow** conversion impls (which must live here because `arrow::*` is foreign and `proto-models` is the only crate where the proto type is local). Optional `pbjson`/`serde`. | `prost`, `arrow`, no datafusion-* crates |
   | `datafusion-common[proto]` | `From`/`TryFrom` impls between proto types and `datafusion-common` types (~25 impls currently in `proto-common`'s `from_proto.rs`/`to_proto.rs`). | adds optional dep on `proto-models` |
   | `datafusion-expr[proto]` | Impls for `WindowFrame`, `WindowFrameBound`, `WindowFrameUnits`, `WriteOp`, `NullTreatment`, `JoinConstraint` (~6 from `datafusion-proto`'s `convert.rs`). | adds optional dep on `proto-models` |
   | `datafusion-datasource[proto]` | Impls for `PartitionedFile`, `FileGroup`, `FileRange`, `FileSinkConfig` (~4 from `datafusion-proto`). | adds optional dep on `proto-models` |
   | `datafusion-datasource-{csv,json,parquet}[proto]` | `JsonSink`, `CsvSink`, `ParquetSink` impls (3, one each). | adds optional dep on `proto-models` |
   | `datafusion-proto` | Encode/decode orchestration only. No `convert.rs`, no `FromProto`/`TryFromProto` traits, no `From` impls. | unchanged |
   | `datafusion-proto-common` | **Deleted**, or shrunk to a thin re-export shim if back-compat matters. | n/a |
   
   Cycle check after the move:
   
   - `datafusion-common[proto] → proto-models → {prost, arrow}` — proto-models has no datafusion deps, no cycle. ✓
   - `datafusion-expr[proto] → proto-models` — same, no cycle. ✓
   - `datafusion-datasource[proto] → proto-models` — same, no cycle. Even with the eventual `physical-expr-common[proto] → proto-models` edge from #21929, there's no path back from proto-models to a datafusion crate. ✓
   - All downstream crates can host their own impls under standard `From`/`TryFrom`.
   
   End state for callers:
   
   ```rust
   // Today (workaround in #21929):
   let y = Y::try_from_proto(&p)?;
   let p = protobuf::X::from_proto(y);
   
   // After this cleanup:
   let y: Y = (&p).try_into()?;
   let p: protobuf::X = y.into();
   ```
   
   ### Why it's a separate PR
   
   The work is mechanical but voluminous:
   
   1. **Make `proto-models` standalone** — drop dep on `proto-common`, add an in-crate `pub mod datafusion_common` to resolve `super::datafusion_common::Foo` references in the prost-generated code, regenerate `pbjson` for both `.datafusion` and `.datafusion_common` packages so the embedded common types get serde impls under the `json` feature.
   2. **Move arrow conversion impls into `proto-models`** — about ~6 impl blocks (Schema, Field, ArrowType, DataType, TimeUnit, IntervalUnit) currently in `proto-common/src/from_proto/mod.rs` and `to_proto/mod.rs`. These must live in `proto-models` because both `arrow::Schema` and `proto::Schema` are foreign to every other crate. Add `arrow` dep.
   3. **Move `datafusion-common` impls** — the remaining ~25 impls in `proto-common`'s from/to_proto modules go into a new `datafusion-common[proto]` module. `datafusion-common` gains an optional `proto-models` dep under the `proto` feature.
   4. **Update `datafusion-ffi`** — currently uses `datafusion_proto_common::from_proto::parse_proto_fields_to_fields` and `datafusion_proto_common::Field`. Migrate to the new locations.
   5. **Update `datafusion-proto`** — its `protobuf` module currently re-exports the explicit list `Schema, ScalarValue, ArrowType, ...` from `proto-common`'s `protobuf_common`, plus a wildcard from `proto-models`. After consolidation it's just `pub use proto_models::generated::datafusion::*` (and the equivalent for the embedded common types). All ~280 callsites that go through `proto-common`'s types via `let s: protobuf::Schema = arrow.try_into()?` continue to compile transparently because `protobuf::Schema` now resolves to a single type.
   6. **Move plan-level impls** — `datafusion-proto/src/convert.rs` deletes; the ~13 impls there relocate to `datafusion-expr[proto]`, `datafusion-datasource[proto]`, and the file-format crates. ~50 callsites in `datafusion-proto` switch from `Y::from_proto(p)` back to `Y::from(p)`.
   7. **Delete `proto-common`** (or stub it as a thin re-export shim and `#[deprecated]` it).
   
   Verified mid-attempt during #21929: the standalone-`proto-models` step works (`cargo check -p datafusion-proto-models --features json` passes), but it surfaces ~280 type mismatches in `datafusion-proto` because of two parallel `Schema` types (`proto-common`'s and `proto-models`'s embedded copy) — those mismatches go away once the consolidation in step 5 is done. So the steps must land together; you can't do step 1 without doing through step 5 in the same PR.
   
   ### Sequencing in a single follow-up PR
   
   Recommended commit shape (4-6 hours of careful work):
   
   1. `Make datafusion-proto-models standalone` — split `datafusion_common` into a sibling module, drop proto-common dep, regenerate pbjson with both packages.
   2. `Move arrow conversion impls into proto-models` — extract from proto-common.
   3. `Add proto feature to datafusion-common, move target-type impls there` — extract the rest from proto-common.
   4. `Migrate datafusion-ffi to the new paths` — small, ~5 imports.
   5. `Use proto-models types throughout datafusion-proto, drop FromProto/TryFromProto` — this is where the ~280 mismatches resolve and `convert.rs` deletes.
   6. `Add proto feature to datafusion-expr/datasource/{csv,json,parquet}, move plan-level impls` — relocates the ~13 impls in datafusion-proto.
   7. `Delete (or stub) datafusion-proto-common` — depending on whether we want a back-compat shim.
   
   Each commit should keep the workspace compiling. Tests stay green throughout because wire format is preserved.
   
   ### Out-of-scope notes
   
   - Wire compatibility is preserved — this is purely a Rust-level refactor, the prost types and serialized bytes are unchanged.
   - The `PhysicalPlanNodeExt` trait introduced in #21929 (for inherent-on-foreign-type orphan-rule cases) is unrelated and stays. Its public-API break (`use datafusion_proto::physical_plan::PhysicalPlanNodeExt;`) is genuinely required.
   - The `to_proto` hook on `PhysicalExpr` (the actual goal of this issue) is delivered in #21929 and unaffected by this follow-up.
   
   cc #21929


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #21835:
URL: https://github.com/apache/datafusion/issues/21835#issuecomment-4319819618

   From the outset, I don't see any blockers. I do think we'd want to make sure we address the window/aggregates as listed in the open questions. I think we need the plan to make sure the CI builds with this feature enabled so that as expressions are added in the core repo we guarantee they have the method implemented.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21835:
URL: https://github.com/apache/datafusion/issues/21835#issuecomment-4363079342

   @alamb having experimented in https://github.com/apache/datafusion/pull/21966 the now somewhat obvious issue with a generic Serde trait approach is that it is incompatible with fixed schema serialization formats like Protobuf. E.g. consider one branch in the trait implementation ads a field and another doesn't. Since there is no static schema it is not possible to form a single valid protobuf model even dynamically.
   
   So if we want this for protobuf (the only current offender as far as I'm concerned) I think we need to go with something like https://github.com/apache/datafusion/pull/21949 and leave a generic Serde approach for later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Split proto serialization to encapsulate private state [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed issue #21835: Split proto serialization to encapsulate private state
URL: https://github.com/apache/datafusion/issues/21835


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org