You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayshrivastava (via GitHub)" <gi...@apache.org> on 2026/08/11 18:43:27 UTC
[I] make HashTableLookupExpr serializable [datafusion]
jayshrivastava opened a new issue, #24277:
URL: https://github.com/apache/datafusion/issues/24277
### Is your feature request related to a problem or challenge?
See https://github.com/apache/datafusion/blob/5d6146ca3fc0c0c601c11bd04ed55e6eb4327dfe/datafusion/proto/tests/cases/roundtrip_physical_plan.rs?plain=1#L3963-L3969
If you take an `ExecutionPlan` and serialize it, all occurrences of `HashTableLookupExpr` are replaced with `lit(true)`. This makes it difficult to send dynamic filters across the network.
### Describe the solution you'd like
Ideally, there's some way to serialize and deserialize these expressions, potentially by serializing the seed
```
message HashTableLookupExpr {
repeated PhysicalExprNode on_columns = 1;
uint64 seed = 2;
repeated fixed64 build_hashes = 3;
}
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make HashTableLookupExpr serializable [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #24277:
URL: https://github.com/apache/datafusion/issues/24277#issuecomment-5498742918
@barbarj are you still working on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make HashTableLookupExpr serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on issue #24277:
URL: https://github.com/apache/datafusion/issues/24277#issuecomment-5259698244
I'll implement this soon, but at high-level, the design I'm thinking is:
### Serialization
For evaluation purposes, we only need the hashes in the `HashMap` case. The protobuf then looks like, with the contents of `ArrayMapNode` to be determined:
```protobuf
message PhysicalHashTableLookupExprNode {
repeated PhysicalExprNode on_columns = 1;
uint64 seed = 2;
string description = 3;
oneof map {
HashSetMapNode hash_map = 4;
ArrayMapNode array_map = 5;
}
}
message HashSetMapNode {
repeated fixed64 build_hashes = 1;
}
```
We add a `hashes` access method:
```rust
pub trait JoinHashMapType {
pub fn hashes(&self) -> Vec<u64>;
}
```
### Deserialization
For the `ArrayMapNode` case, we just deserialize to the `ArrayMap` variant of `HashTableLookupExpr`. For the `HashSetMapNode` case, we add a new impl of `JoinHashMapType` with `contain_hashes`/`len`/`is_empty` implemented. It will return internal errors from the build-side mutation and get_matched_indices methods. This will allow it be evaluated after deserialization but not updated. (Will document that this should be constructed during deserialization only and is an immutable snapshot of the expression).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make HashTableLookupExpr serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on issue #24277:
URL: https://github.com/apache/datafusion/issues/24277#issuecomment-5499524111
> [@barbarj](https://github.com/barbarj) are you still working on this?
Yeah! I was on vacation last week, hence the inactivity. I'll be following up on that #24382 shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make HashTableLookupExpr serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on issue #24277:
URL: https://github.com/apache/datafusion/issues/24277#issuecomment-5259031533
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org