You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "barbarj (via GitHub)" <gi...@apache.org> on 2026/08/14 20:37:39 UTC
[PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
barbarj opened a new pull request, #24382:
URL: https://github.com/apache/datafusion/pull/24382
## Which issue does this PR close?
- Closes #24277
## Rationale for this change
In order for DataFusion Distributed (and presumably other distributed DF projects) to make use of `HashTableLookupExpr` as a dynamic filter across network boundaries, it needs to be serializable.
## What changes are included in this PR?
The only sticky part of serializing `HashTableLookupExpr` is its `Map`. The inner members of the two variants of `Map`, `HashMap` and `ArrayMap` both support much more functionality than is needed to evaluate this as a dynamic expression. So, in order to simply serialization and minimize the on-wire size, we serialize only the aspects required for expression evaluation (i.e. membership checks)
We replace `HashTableLookupExpr`'s `Map` with a local version that includes the membership-only variants. These are implemented such that they are only constructible via deserialization.
## Are these changes tested?
The `roundtrip_hash_table_lookup_expr_to_lit` test is replaced with two regular roundtrip tests, one each for the `HashMap` and `ArrayMap` versions of `HashTableLookupExpr`.
This PR also adds a bunch of tests testing the post-deserialization behavior of the new membership-only variants.
## Are there any user-facing changes?
- `JoinHashMapType` has a new required method `hashes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5499599784
@adriangb or @zhuqi-lucas Can you take a look at this? Do you have any thoughts about this approach vs waiting for bloom filters to ship with #24768 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5298149069
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24382?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `63.46154%` with `266 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.15%. Comparing base ([`30ca39b`](https://app.codecov.io/gh/apache/datafusion/commit/30ca39bef9ea2e746e143727f9de48b1608edc2a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e4ce61b`](https://app.codecov.io/gh/apache/datafusion/commit/e4ce61bb0600987f1b6847c9fe45c0cac6dff787?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 46 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24382?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/proto-models/src/generated/pbjson.rs](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&filepath=datafusion%2Fproto-models%2Fsrc%2Fgenerated%2Fpbjson.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by1tb2RlbHMvc3JjL2dlbmVyYXRlZC9wYmpzb24ucnM=) | 0.00% | [234 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...-plan/src/joins/hash\_join/partitioned\_hash\_eval.rs](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fhash_join%2Fpartitioned_hash_eval.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9oYXNoX2pvaW4vcGFydGl0aW9uZWRfaGFzaF9ldmFsLnJz) | 93.81% | [22 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...usion/physical-plan/src/joins/stream\_join\_utils.rs](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fstream_join_utils.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9zdHJlYW1fam9pbl91dGlscy5ycw==) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/proto/src/physical\_plan/from\_proto.rs](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&filepath=datafusion%2Fproto%2Fsrc%2Fphysical_plan%2Ffrom_proto.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9wcm90by9zcmMvcGh5c2ljYWxfcGxhbi9mcm9tX3Byb3RvLnJz) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24382?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24382 +/- ##
==========================================
- Coverage 81.29% 81.15% -0.14%
==========================================
Files 1110 1110
Lines 385168 389445 +4277
Branches 385168 389445 +4277
==========================================
+ Hits 313111 316072 +2961
- Misses 53583 54775 +1192
- Partials 18474 18598 +124
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24382?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5300029022
@jayshrivastava, @barbarj, @adriangb: The deciding line between `List` and `Hash` is already configurable: relatively small sets get list, larger sets get `Hash`.
Given that, should the presence of a `Hash` like this already suggest that systems which care about serializing a `Hash` should probably be using a bloom filter instead? Or is the thinking that there are multiple size thresholds, and `Hash` needs to be serializable for the middle one?:
* small: `List`
* medium: `Hash`
* large: "bloom"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5300417282
I think the main thought behind the original design and split was that locally at least there's no point in building a Bloom filter if you already have a hash table in memory. It makes sense to me that if you want to serialize across the wire the tradeoff is very different, a bloom filter would be better. I'm not sure if that means you would want to build a Bloom filter upfront or build it when you serialize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5499796478
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-physical-plan v55.0.0 (current)
Built [ 56.220s] (current)
Parsing datafusion-physical-plan v55.0.0 (current)
Parsed [ 0.140s] (current)
Building datafusion-physical-plan v55.0.0 (baseline)
Built [ 36.680s] (baseline)
Parsing datafusion-physical-plan v55.0.0 (baseline)
Parsed [ 0.142s] (baseline)
Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.980s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure trait_method_added: pub trait method added ---
Description:
A non-sealed public trait added a new method without a default implementation, which breaks downstream implementations of the trait
ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-item-no-default
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_method_added.ron
Failed in:
trait method datafusion_physical_plan::joins::utils::JoinHashMapType::hashes in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/join_hash_map.rs:151
trait method datafusion_physical_plan::joins::join_hash_map::JoinHashMapType::hashes in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/join_hash_map.rs:151
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 96.664s] datafusion-physical-plan
Building datafusion-proto v55.0.0 (current)
Built [ 51.434s] (current)
Parsing datafusion-proto v55.0.0 (current)
Parsed [ 0.018s] (current)
Building datafusion-proto v55.0.0 (baseline)
Built [ 52.802s] (baseline)
Parsing datafusion-proto v55.0.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.148s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 106.103s] datafusion-proto
Building datafusion-proto-models v55.0.0 (current)
Built [ 24.675s] (current)
Parsing datafusion-proto-models v55.0.0 (current)
Parsed [ 0.121s] (current)
Building datafusion-proto-models v55.0.0 (baseline)
Built [ 23.958s] (baseline)
Parsing datafusion-proto-models v55.0.0 (baseline)
Parsed [ 0.120s] (baseline)
Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 2.409s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant ExprType:HashTableLookupExpr in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1634
variant ExprType:HashTableLookupExpr in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1634
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 52.674s] datafusion-proto-models
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5499853985
I'm very much 👎🏻 on serializing the in memory map across the wire.
I'm already considering a proposal to disable them by default because the current implementations are slower than no filter if the joint is not selective. Adding more additional work that happens just because the filter exists will only make this worse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5500215771
No, it would just turn it off by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5337818375
My intent here was to keep the `List`/`Hash` choice across the network, though I can see a case for bumping the `Hash` to "bloom" when serializing to keep the wire size down. But I'd expect we'd need another intra-DataFusion reason to add the bloom filter code besides this.
Related: #16435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "stuhood (via GitHub)" <gi...@apache.org>.
stuhood commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5463511765
Related: bloom filters coming to spillable hash joins: https://github.com/apache/datafusion/issues/24768
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5297985959
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion-physical-plan v54.1.0 (current)
Built [ 43.439s] (current)
Parsing datafusion-physical-plan v54.1.0 (current)
Parsed [ 0.154s] (current)
Building datafusion-physical-plan v54.1.0 (baseline)
Built [ 38.331s] (baseline)
Parsing datafusion-physical-plan v54.1.0 (baseline)
Parsed [ 0.151s] (baseline)
Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 1.055s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure trait_method_added: pub trait method added ---
Description:
A non-sealed public trait added a new method without a default implementation, which breaks downstream implementations of the trait
ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-item-no-default
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_method_added.ron
Failed in:
trait method datafusion_physical_plan::joins::utils::JoinHashMapType::hashes in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/join_hash_map.rs:151
trait method datafusion_physical_plan::joins::join_hash_map::JoinHashMapType::hashes in file /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/joins/join_hash_map.rs:151
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 85.382s] datafusion-physical-plan
Building datafusion-proto v54.1.0 (current)
Built [ 55.101s] (current)
Parsing datafusion-proto v54.1.0 (current)
Parsed [ 0.019s] (current)
Building datafusion-proto v54.1.0 (baseline)
Built [ 56.833s] (baseline)
Parsing datafusion-proto v54.1.0 (baseline)
Parsed [ 0.021s] (baseline)
Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.157s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 113.589s] datafusion-proto
Building datafusion-proto-models v54.1.0 (current)
Built [ 26.620s] (current)
Parsing datafusion-proto-models v54.1.0 (current)
Parsed [ 0.137s] (current)
Building datafusion-proto-models v54.1.0 (baseline)
Built [ 26.163s] (baseline)
Parsing datafusion-proto-models v54.1.0 (baseline)
Parsed [ 0.143s] (baseline)
Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 2.428s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant ExprType:HashTableLookupExpr in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1626
variant ExprType:HashTableLookupExpr in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1626
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 57.464s] datafusion-proto-models
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on code in PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#discussion_r3787413274
##########
docs/source/library-user-guide/upgrading/55.0.0.md:
##########
@@ -1291,3 +1219,28 @@ let table_opts = TableParquetOptions::try_from(&proto_table_opts)?;
```
See [issue #24019](https://github.com/apache/datafusion/issues/24019) for details.
+
+### `JoinHashMapType` has a new required method `hashes`
Review Comment:
If this doesn't make it into 55.0, this will need to be moved to 55.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): Make `HashTableLookupExpr` serializable [datafusion]
Posted by "barbarj (via GitHub)" <gi...@apache.org>.
barbarj commented on PR #24382:
URL: https://github.com/apache/datafusion/pull/24382#issuecomment-5500012953
> I'm very much 👎🏻 on serializing the in memory map across the wire.
>
> I'm already considering a proposal to disable them by default because the current implementations are slower than no filter if the joint is not selective. Adding more additional work that happens just because the filter exists will only make this worse.
Noted! That's instructive. If implemented, that proposal would disable the use of `HashTableLookupExpr`? Is that right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org