You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nanookclaw (via GitHub)" <gi...@apache.org> on 2026/05/24 19:51:44 UTC
[PR] refactor: port HashExpr proto hooks [datafusion]
nanookclaw opened a new pull request, #22502:
URL: https://github.com/apache/datafusion/pull/22502
## Which issue does this PR close?
- Closes #22432.
## Rationale for this change
`HashExpr` is part of the physical expression proto cleanup tracked by #22418. Its protobuf serialization should live with the expression implementation instead of in the central proto downcast chains, matching the hook pattern added in #21929.
## What changes are included in this PR?
This adds `HashExpr`'s `PhysicalExpr::try_to_proto` override and a feature-gated inherent `HashExpr::try_from_proto`. The existing `PhysicalHashExprNode` wire shape is preserved, including `expr_id: None`, while `datafusion-proto` now routes `ExprType::HashExpr` decoding through the hook and no longer has a central `HashExpr` serialization arm. `datafusion-physical-plan` now exposes a `proto` feature so the hook code only compiles when proto support is requested.
## Are these changes tested?
Yes. Added focused direct hook tests for encoding, decoding, and rejecting a wrong `expr_type`.
Commands run:
- `cargo fmt --all -- --check`
- `cargo check -p datafusion-physical-plan`
- `cargo test -p datafusion-physical-plan --features proto hash_expr_try`
- `cargo test -p datafusion-proto --test proto_integration cases::roundtrip_physical_plan::roundtrip_hash_expr`
- `cargo check -p datafusion-proto`
- `cargo clippy -p datafusion-physical-plan --features proto --all-targets -- -D warnings`
- `cargo clippy -p datafusion-proto --all-targets -- -D warnings`
## Are there any user-facing changes?
No. This preserves the existing protobuf representation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #22502:
URL: https://github.com/apache/datafusion/pull/22502#issuecomment-4535277236
@nanookclaw could you rebase and resolve conflicts please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #22502:
URL: https://github.com/apache/datafusion/pull/22502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "nanookclaw (via GitHub)" <gi...@apache.org>.
nanookclaw commented on PR #22502:
URL: https://github.com/apache/datafusion/pull/22502#issuecomment-4536133512
Rebased and resolved the conflicts. The new head is `768f68d50`.
Validation run after the rebase:
- `cargo fmt --check -- datafusion/proto/src/physical_plan/from_proto.rs datafusion/proto/src/physical_plan/to_proto.rs`
- `cargo test -p datafusion-physical-plan hash_expr_try --features proto --lib` → 3 passed
- `cargo check -p datafusion-proto`
- `git diff --check upstream/main...HEAD`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "nanookclaw (via GitHub)" <gi...@apache.org>.
nanookclaw commented on PR #22502:
URL: https://github.com/apache/datafusion/pull/22502#issuecomment-4560762912
Updated this to use the PR #22513 helpers for `HashExpr` proto children in `7d14ebc43`:
- `ctx.encode_children_expressions(&self.on_columns)?`
- `ctx.decode_children_expressions(&hash_expr.on_columns)?`
Validation:
- `cargo fmt -- datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs`
- `git diff --check`
- `cargo test -p datafusion-physical-plan --features proto hash_expr_try_ --lib` (3 passed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "nanookclaw (via GitHub)" <gi...@apache.org>.
nanookclaw commented on PR #22502:
URL: https://github.com/apache/datafusion/pull/22502#issuecomment-4548182964
Rebased `fix/hash-expr-proto-hooks` onto current `main` and force-pushed head `20c2832988511969334e9ec4b189e1952365c3d3`.
Conflict resolution kept upstream's `HashTableLookupExpr` proto hook behavior and kept this PR's `HashExpr` hook migration, so `HashExpr` now goes through `try_to_proto` / `try_from_proto` rather than the central `to_proto` downcast branch.
Local validation passed:
- `cargo fmt --check -- datafusion/proto/src/physical_plan/from_proto.rs datafusion/proto/src/physical_plan/to_proto.rs datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs`
- `cargo test -p datafusion-physical-plan hash_expr_try --features proto --lib`
- `cargo check -p datafusion-proto`
- `git diff --check upstream/main...HEAD`
GitHub checks have restarted on the new head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22502:
URL: https://github.com/apache/datafusion/pull/22502#discussion_r3311499659
##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -199,6 +201,63 @@ impl PhysicalExpr for HashExpr {
fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.description)
}
+
+ #[cfg(feature = "proto")]
+ fn try_to_proto(
+ &self,
+ ctx: &datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
+ ) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
+ use datafusion_proto_models::protobuf;
+ let on_columns = self
+ .on_columns
+ .iter()
+ .map(|e| ctx.encode_child(e))
+ .collect::<Result<Vec<_>>>()?;
+ Ok(Some(protobuf::PhysicalExprNode {
+ expr_id: None,
+ expr_type: Some(protobuf::physical_expr_node::ExprType::HashExpr(
+ protobuf::PhysicalHashExprNode {
+ on_columns,
+ seed0: self.seed(),
+ description: self.description.clone(),
+ },
+ )),
+ }))
+ }
+}
+
+#[cfg(feature = "proto")]
+impl HashExpr {
+ /// Reconstruct a [`HashExpr`] from its protobuf representation.
+ ///
+ /// Takes the whole [`PhysicalExprNode`], the exact inverse of what
+ /// [`PhysicalExpr::try_to_proto`] produces, so every expression's
+ /// `try_from_proto` shares one signature. Child sub-expressions are
+ /// decoded recursively via [`PhysicalExprDecodeCtx::decode`].
+ ///
+ /// [`PhysicalExprNode`]: datafusion_proto_models::protobuf::PhysicalExprNode
+ /// [`PhysicalExpr::try_to_proto`]: datafusion_physical_expr_common::physical_expr::PhysicalExpr::try_to_proto
+ /// [`PhysicalExprDecodeCtx::decode`]: datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx::decode
+ pub fn try_from_proto(
+ node: &datafusion_proto_models::protobuf::PhysicalExprNode,
+ ctx: &datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx<'_>,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ use datafusion_proto_models::protobuf;
+ let hash_expr = match &node.expr_type {
+ Some(protobuf::physical_expr_node::ExprType::HashExpr(h)) => h,
+ _ => return internal_err!("PhysicalExprNode is not a HashExpr"),
+ };
Review Comment:
Can we use the helpers from https://github.com/apache/datafusion/pull/22513?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: port HashExpr proto hooks [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22502:
URL: https://github.com/apache/datafusion/pull/22502#discussion_r3318695601
##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -498,6 +549,172 @@ mod tests {
assert_eq!(compute_hash(&expr1), compute_hash(&expr2));
}
+ #[cfg(feature = "proto")]
+ mod proto_tests {
+ use super::*;
+ use arrow::datatypes::{DataType, Field};
+ use datafusion_common::internal_datafusion_err;
+ use datafusion_physical_expr_common::physical_expr::proto_decode::{
+ PhysicalExprDecode, PhysicalExprDecodeCtx,
+ };
+ use datafusion_physical_expr_common::physical_expr::proto_encode::{
+ PhysicalExprEncode, PhysicalExprEncodeCtx,
+ };
+ use datafusion_proto_models::protobuf;
+
+ struct TestEncoder;
+
+ impl PhysicalExprEncode for TestEncoder {
+ fn encode(
+ &self,
+ expr: &Arc<dyn PhysicalExpr>,
+ ) -> Result<protobuf::PhysicalExprNode> {
+ let ctx = PhysicalExprEncodeCtx::new(self);
+ expr.try_to_proto(&ctx)?.ok_or_else(|| {
+ internal_datafusion_err!("test encoder cannot encode {expr:?}")
+ })
+ }
+ }
+
+ struct TestDecoder;
+
+ impl PhysicalExprDecode for TestDecoder {
+ fn decode(
+ &self,
+ node: &protobuf::PhysicalExprNode,
+ schema: &Schema,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ let ctx = PhysicalExprDecodeCtx::new(schema, self);
+ match &node.expr_type {
+ Some(protobuf::physical_expr_node::ExprType::Column(_)) => {
+ Column::try_from_proto(node, &ctx)
+ }
+ _ => internal_err!("test decoder cannot decode {node:?}"),
+ }
+ }
+ }
+
+ fn test_decode_ctx<'a>(
+ schema: &'a Schema,
+ decoder: &'a TestDecoder,
+ ) -> PhysicalExprDecodeCtx<'a> {
+ PhysicalExprDecodeCtx::new(schema, decoder)
+ }
Review Comment:
Should we put these into https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/proto_test_util.rs ?
##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -199,6 +201,55 @@ impl PhysicalExpr for HashExpr {
fn fmt_sql(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.description)
}
+
+ #[cfg(feature = "proto")]
+ fn try_to_proto(
+ &self,
+ ctx: &datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
+ ) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
+ use datafusion_proto_models::protobuf;
+ let on_columns = ctx.encode_children_expressions(&self.on_columns)?;
+ Ok(Some(protobuf::PhysicalExprNode {
+ expr_id: None,
+ expr_type: Some(protobuf::physical_expr_node::ExprType::HashExpr(
+ protobuf::PhysicalHashExprNode {
+ on_columns,
+ seed0: self.seed(),
+ description: self.description.clone(),
+ },
+ )),
+ }))
+ }
+}
+
+#[cfg(feature = "proto")]
+impl HashExpr {
+ /// Reconstruct a [`HashExpr`] from its protobuf representation.
+ ///
+ /// Takes the whole [`PhysicalExprNode`], the exact inverse of what
+ /// [`PhysicalExpr::try_to_proto`] produces, so every expression's
+ /// `try_from_proto` shares one signature. Child sub-expressions are
+ /// decoded recursively via [`PhysicalExprDecodeCtx::decode`].
+ ///
+ /// [`PhysicalExprNode`]: datafusion_proto_models::protobuf::PhysicalExprNode
+ /// [`PhysicalExpr::try_to_proto`]: datafusion_physical_expr_common::physical_expr::PhysicalExpr::try_to_proto
+ /// [`PhysicalExprDecodeCtx::decode`]: datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx::decode
+ pub fn try_from_proto(
+ node: &datafusion_proto_models::protobuf::PhysicalExprNode,
+ ctx: &datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx<'_>,
+ ) -> Result<Arc<dyn PhysicalExpr>> {
+ use datafusion_proto_models::protobuf;
+ let hash_expr = match &node.expr_type {
+ Some(protobuf::physical_expr_node::ExprType::HashExpr(h)) => h,
+ _ => return internal_err!("PhysicalExprNode is not a HashExpr"),
+ };
Review Comment:
it might make sense to add a helper for this to `PhysicalExprDecodeCtx` so we also get unified error messages, etc.
##########
datafusion/physical-plan/Cargo.toml:
##########
@@ -39,14 +39,18 @@ workspace = true
[features]
force_hash_collisions = []
-test_utils = ["arrow/test_utils"]
-tokio_coop = []
-tokio_coop_fallback = []
+# Enables `PhysicalExpr::try_to_proto` / `try_from_proto` hooks on the
+# physical expressions defined in this crate (e.g. `HashExpr`). Off by
+# default so consumers that never serialize plans pay nothing.
proto = [
"dep:datafusion-proto-models",
"dep:datafusion-proto-common",
+ "datafusion-physical-expr/proto",
"datafusion-physical-expr-common/proto",
]
+test_utils = ["arrow/test_utils"]
+tokio_coop = []
+tokio_coop_fallback = []
Review Comment:
I guess a linter auto sorted these but can we move them back up to avoid conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org