You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/02/05 08:56:26 UTC

[I] Implement Protobuf Support for CastExpr and CastColumnExpr [datafusion]

kosiew opened a new issue, #20165:
URL: https://github.com/apache/datafusion/issues/20165

   Add protobuf serialization and deserialization support for the unified `CastExpr` (logical) and `CastColumnExpr` (physical) to enable distributed execution, Flight SQL, CLI round‑trips, and cross-process plan exchange.
   
   ## Scope
   
   Implement complete serialization/deserialization round‑trip support in `datafusion/proto/` for both logical and physical cast expressions.
   
   ### Tasks
   
   **Logical Cast Serialization:**
   - [ ] Update `datafusion/proto/proto/logical_plan.proto` to support enhanced `CastExpr`
   - [ ] Add proto serde for enhanced `CastExpr` in `datafusion/proto/src/logical_plan.rs`
   - [ ] Implement round-trip tests for logical cast expressions
   
   **Physical Cast Serialization:**
   - [ ] Add proto message for `CastColumnExpr` in `datafusion/proto/proto/physical_plan.proto`
   - [ ] Implement serialization logic (`PhysicalExpr` → protobuf)
   - [ ] Implement deserialization logic (protobuf → `PhysicalExpr`)
   - [ ] Add round-trip tests for `CastColumnExpr` variants
   - [ ] Test with distributed/Flight SQL scenarios
   
   **Integration & Testing:**
   - [ ] Validate round-trip equality (expr → proto → expr)
   - [ ] Test with struct casting scenarios
   - [ ] Ensure backward compatibility (if applicable)
   - [ ] Add comprehensive proto round-trip test suite
   
   ### Files/Modules
   
   - `datafusion/proto/proto/logical_plan.proto` (logical cast message)
   - `datafusion/proto/proto/physical_plan.proto` (physical cast message)
   - `datafusion/proto/src/logical_plan.rs` (logical cast serde)
   - `datafusion/proto/src/physical_expr.rs` (physical cast serde)
   - Proto-related integration tests
   
   ### Key Features
   
   - **Lossless round-trip:** Serialized and deserialized expressions are equivalent
   - **Struct awareness:** Serialization preserves struct field casting semantics
   - **Nullability propagation:** Field-level nullability metadata survives round-trip
   - **Backward compatibility:** Graceful handling of older proto versions (if applicable)
   - **Clear error messages:** Helpful errors when deserialization fails
   
   ### Acceptance Criteria
   
   ✅ Protobuf messages defined for both logical and physical cast expressions  
   ✅ Serialization/deserialization code is complete and tested  
   ✅ Round-trip tests pass for all `CastExpr` and `CastColumnExpr` variants  
   ✅ Struct casting scenarios work end-to-end with serialization  
   ✅ Code compiles and all proto tests pass  
   ✅ Documented any proto breaking changes or version notes  
   ✅ Integration tests validate distributed/Flight SQL compatibility  
   
   ## Notes
   
   - **Scope:** This PR enables distributed execution, Flight SQL, and CLI serialization for the cast expressions introduced in #20164.
   - **Breaking changes:** Proto updates may require versioning or migration notes if deployed to existing systems.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement Protobuf Support for CastExpr and CastColumnExpr [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew closed issue #20165: Implement Protobuf Support for CastExpr and CastColumnExpr
URL: https://github.com/apache/datafusion/issues/20165


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Implement Protobuf Support for CastExpr and CastColumnExpr [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #20165:
URL: https://github.com/apache/datafusion/issues/20165#issuecomment-3859596377

   Proto changes cannot be decoupled into its own PR.
   
   The proto changes have to be implemented together with CastColumnExpr as it is part of the physical expression tree, which is serialized/deserialized in datafusion-proto. Once we add a new physical expression variant or change its data model (like OwnedCastOptions, schema-aware fields, etc.), we need to update the proto layer so:
   
   1. Plans can be serialized/deserialized correctly
   CastColumnExpr gets sent over the wire (or stored) when physical plans are serialized. Without proto updates, roundtrip tests fail or the plan can’t be restored.
   
   2. Owned cast/format options now need to be preserved
   The new design stores options in owned Strings. That means the protobuf needs fields for PhysicalCastOptions and FormatOptions, plus the corresponding conversion logic in from_proto.rs and to_proto.rs.
   
   3. Round-trip tests assert fidelity
   The roundtrip_physical_plan.rs tests exercise proto serialization, so those must be updated whenever expression structs or option handling changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org