You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/07/28 17:36:19 UTC
[PR] Enable physical filter pushdown for hash joins [datafusion]
adriangb opened a new pull request, #16954:
URL: https://github.com/apache/datafusion/pull/16954
Replaces a piece of https://github.com/apache/datafusion/pull/16445 to break up into smaller PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16954:
URL: https://github.com/apache/datafusion/pull/16954#issuecomment-3129333785
๐ค: Benchmark completed
<details><summary>Details</summary>
<p>
```
Comparing HEAD and hash-join-allow-pushdown
--------------------
Benchmark clickbench_extended.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Query โ HEAD โ hash-join-allow-pushdown โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 1934.91 ms โ 1967.56 ms โ no change โ
โ QQuery 1 โ 680.09 ms โ 686.86 ms โ no change โ
โ QQuery 2 โ 1343.67 ms โ 1367.13 ms โ no change โ
โ QQuery 3 โ 671.54 ms โ 670.78 ms โ no change โ
โ QQuery 4 โ 1343.96 ms โ 1369.05 ms โ no change โ
โ QQuery 5 โ 14659.89 ms โ 14644.36 ms โ no change โ
โ QQuery 6 โ 2072.69 ms โ 2065.36 ms โ no change โ
โ QQuery 7 โ 1921.02 ms โ 1962.28 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 24627.76ms โ
โ Total Time (hash-join-allow-pushdown) โ 24733.38ms โ
โ Average Time (HEAD) โ 3078.47ms โ
โ Average Time (hash-join-allow-pushdown) โ 3091.67ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 0 โ
โ Queries with No Change โ 8 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark clickbench_partitioned.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ hash-join-allow-pushdown โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 0 โ 2.33 ms โ 2.75 ms โ 1.18x slower โ
โ QQuery 1 โ 33.29 ms โ 34.53 ms โ no change โ
โ QQuery 2 โ 80.43 ms โ 80.99 ms โ no change โ
โ QQuery 3 โ 98.30 ms โ 96.81 ms โ no change โ
โ QQuery 4 โ 598.15 ms โ 609.25 ms โ no change โ
โ QQuery 5 โ 848.89 ms โ 853.05 ms โ no change โ
โ QQuery 6 โ 2.24 ms โ 2.26 ms โ no change โ
โ QQuery 7 โ 37.45 ms โ 38.15 ms โ no change โ
โ QQuery 8 โ 848.13 ms โ 856.08 ms โ no change โ
โ QQuery 9 โ 1180.14 ms โ 1200.91 ms โ no change โ
โ QQuery 10 โ 257.14 ms โ 259.27 ms โ no change โ
โ QQuery 11 โ 292.14 ms โ 286.57 ms โ no change โ
โ QQuery 12 โ 877.62 ms โ 882.69 ms โ no change โ
โ QQuery 13 โ 1234.54 ms โ 1242.03 ms โ no change โ
โ QQuery 14 โ 838.08 ms โ 828.27 ms โ no change โ
โ QQuery 15 โ 774.78 ms โ 792.96 ms โ no change โ
โ QQuery 16 โ 1668.70 ms โ 1625.33 ms โ no change โ
โ QQuery 17 โ 1625.72 ms โ 1618.61 ms โ no change โ
โ QQuery 18 โ 2860.56 ms โ 2894.43 ms โ no change โ
โ QQuery 19 โ 88.99 ms โ 87.42 ms โ no change โ
โ QQuery 20 โ 1195.22 ms โ 1179.02 ms โ no change โ
โ QQuery 21 โ 1334.91 ms โ 1336.30 ms โ no change โ
โ QQuery 22 โ 2194.96 ms โ 2217.46 ms โ no change โ
โ QQuery 23 โ 7536.86 ms โ 7550.03 ms โ no change โ
โ QQuery 24 โ 443.15 ms โ 446.15 ms โ no change โ
โ QQuery 25 โ 315.38 ms โ 310.61 ms โ no change โ
โ QQuery 26 โ 435.79 ms โ 441.80 ms โ no change โ
โ QQuery 27 โ 1562.84 ms โ 1568.25 ms โ no change โ
โ QQuery 28 โ 12113.23 ms โ 12088.54 ms โ no change โ
โ QQuery 29 โ 530.52 ms โ 526.00 ms โ no change โ
โ QQuery 30 โ 801.92 ms โ 803.02 ms โ no change โ
โ QQuery 31 โ 791.77 ms โ 798.40 ms โ no change โ
โ QQuery 32 โ 2419.43 ms โ 2478.76 ms โ no change โ
โ QQuery 33 โ 3197.37 ms โ 3205.40 ms โ no change โ
โ QQuery 34 โ 3223.48 ms โ 3259.30 ms โ no change โ
โ QQuery 35 โ 1252.88 ms โ 1242.37 ms โ no change โ
โ QQuery 36 โ 121.83 ms โ 122.84 ms โ no change โ
โ QQuery 37 โ 52.28 ms โ 54.31 ms โ no change โ
โ QQuery 38 โ 121.19 ms โ 122.45 ms โ no change โ
โ QQuery 39 โ 199.03 ms โ 197.62 ms โ no change โ
โ QQuery 40 โ 42.09 ms โ 42.93 ms โ no change โ
โ QQuery 41 โ 38.03 ms โ 39.56 ms โ no change โ
โ QQuery 42 โ 33.06 ms โ 32.59 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 54204.85ms โ
โ Total Time (hash-join-allow-pushdown) โ 54356.11ms โ
โ Average Time (HEAD) โ 1260.58ms โ
โ Average Time (hash-join-allow-pushdown) โ 1264.10ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 42 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโ
--------------------
Benchmark tpch_mem_sf1.json
--------------------
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโ
โ Query โ HEAD โ hash-join-allow-pushdown โ Change โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ QQuery 1 โ 95.09 ms โ 101.86 ms โ 1.07x slower โ
โ QQuery 2 โ 20.99 ms โ 20.09 ms โ no change โ
โ QQuery 3 โ 33.04 ms โ 32.87 ms โ no change โ
โ QQuery 4 โ 18.26 ms โ 18.58 ms โ no change โ
โ QQuery 5 โ 49.55 ms โ 49.54 ms โ no change โ
โ QQuery 6 โ 11.91 ms โ 12.02 ms โ no change โ
โ QQuery 7 โ 86.50 ms โ 88.81 ms โ no change โ
โ QQuery 8 โ 24.43 ms โ 25.01 ms โ no change โ
โ QQuery 9 โ 53.68 ms โ 53.40 ms โ no change โ
โ QQuery 10 โ 42.33 ms โ 42.27 ms โ no change โ
โ QQuery 11 โ 11.51 ms โ 11.07 ms โ no change โ
โ QQuery 12 โ 34.99 ms โ 35.18 ms โ no change โ
โ QQuery 13 โ 25.84 ms โ 26.41 ms โ no change โ
โ QQuery 14 โ 9.70 ms โ 9.70 ms โ no change โ
โ QQuery 15 โ 18.93 ms โ 19.36 ms โ no change โ
โ QQuery 16 โ 18.21 ms โ 18.01 ms โ no change โ
โ QQuery 17 โ 96.23 ms โ 97.18 ms โ no change โ
โ QQuery 18 โ 198.58 ms โ 191.80 ms โ no change โ
โ QQuery 19 โ 25.71 ms โ 24.59 ms โ no change โ
โ QQuery 20 โ 30.84 ms โ 31.70 ms โ no change โ
โ QQuery 21 โ 145.91 ms โ 142.65 ms โ no change โ
โ QQuery 22 โ 15.11 ms โ 14.55 ms โ no change โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโ
โ Benchmark Summary โ โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ Total Time (HEAD) โ 1067.31ms โ
โ Total Time (hash-join-allow-pushdown) โ 1066.65ms โ
โ Average Time (HEAD) โ 48.51ms โ
โ Average Time (hash-join-allow-pushdown) โ 48.48ms โ
โ Queries Faster โ 0 โ
โ Queries Slower โ 1 โ
โ Queries with No Change โ 21 โ
โ Queries with Failure โ 0 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโ
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #16954:
URL: https://github.com/apache/datafusion/pull/16954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #16954:
URL: https://github.com/apache/datafusion/pull/16954#issuecomment-3128992040
> Results will be posted here when complete
FWIW my expectation is that this won't make much of a difference unless there is a chain like TopK -> HashJoin -> Scan and filter pushdown into the scan is enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #16954:
URL: https://github.com/apache/datafusion/pull/16954#discussion_r2240550577
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -944,6 +949,53 @@ impl ExecutionPlan for HashJoinExec {
try_embed_projection(projection, self)
}
}
+
+ fn gather_filters_for_pushdown(
+ &self,
+ phase: FilterPushdownPhase,
+ parent_filters: Vec<Arc<dyn PhysicalExpr>>,
+ _config: &ConfigOptions,
+ ) -> Result<FilterDescription> {
+ if self.join_type != JoinType::Inner {
Review Comment:
I recommend we file a ticket and then add a reference to that ticket in the code a ticket reference so someone in the future who stumbles on this code can locate the relevant context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16954:
URL: https://github.com/apache/datafusion/pull/16954#issuecomment-3128926114
๐ค `./gh_compare_branch.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/gh_compare_branch.sh) Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing hash-join-allow-pushdown (2f76a91a388bb82268a2fe2a4d826b41a8c01bf5) to c6d55207161e400e53645d5ee7d7bf16cd024c2f [diff](https://github.com/apache/datafusion/compare/c6d55207161e400e53645d5ee7d7bf16cd024c2f..2f76a91a388bb82268a2fe2a4d826b41a8c01bf5) using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16954:
URL: https://github.com/apache/datafusion/pull/16954#issuecomment-3128933148
I will try and review this PR tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Enable physical filter pushdown for hash joins [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #16954:
URL: https://github.com/apache/datafusion/pull/16954#discussion_r2241538423
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -944,6 +949,53 @@ impl ExecutionPlan for HashJoinExec {
try_embed_projection(projection, self)
}
}
+
+ fn gather_filters_for_pushdown(
+ &self,
+ phase: FilterPushdownPhase,
+ parent_filters: Vec<Arc<dyn PhysicalExpr>>,
+ _config: &ConfigOptions,
+ ) -> Result<FilterDescription> {
+ if self.join_type != JoinType::Inner {
Review Comment:
https://github.com/apache/datafusion/issues/16973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org