You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "gongxun0928 (via GitHub)" <gi...@apache.org> on 2025/08/22 04:45:08 UTC
[PR] feat: support runtime filter push down to Table AM [cloudberry]
gongxun0928 opened a new pull request, #1324:
URL: https://github.com/apache/cloudberry/pull/1324
when seq scan begins, check whether the scanflags of table am is set to determine whether the runtime filter is pushed down.
When the runtime filter is pushed down to pax am, pax am converts the min/max scankey in the runtime filter into PFTNode and performs min/max filtering.
```
CREATE TABLE t1(c1 int, c2 int, c3 int, c4 int, c5 int) using pax with(minmax_columns='c1,c2');
insert into t1 select i,i,i,i,i from generate_series(1,10000000) i;
analyze t1;
CREATE TABLE t2(c1 int, c2 int, c3 int, c4 int, c5 int) with (appendonly=true, orientation=column) distributed REPLICATED;
INSERT INTO t2 VALUES (1,1,1,1,1), (2,2,2,2,2), (3,3,3,3,3), (4,4,4,4,4);
INSERT INTO t2 select * FROM t2;
INSERT INTO t2 select * FROM t2;
INSERT INTO t2 select * FROM t2;
set gp_enable_runtime_filter_pushdown to off;
EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
SELECT t1.c3 FROM t1, t2 WHERE t1.c2 = t2.c2;
QUERY PLAN
----------------------------------------------------------------------------------
Gather Motion 1:1 (slice1; segments: 1) (actual rows=32 loops=1)
-> Hash Join (actual rows=32 loops=1)
Hash Cond: (t1.c2 = t2.c2)
Extra Text: Hash chain length 8.0 avg, 8 max, using 4 of 524288 buckets.
-> Seq Scan on t1 (actual rows=10000000 loops=1)
-> Hash (actual rows=32 loops=1)
Buckets: 524288 Batches: 1 Memory Usage: 4098kB
-> Seq Scan on t2 (actual rows=32 loops=1)
Optimizer: GPORCA
(9 rows)
Time: 1576.548 ms (00:01.577)
gpadmin=# set gp_enable_runtime_filter_pushdown to on;
SET
Time: 0.362 ms
gpadmin=# EXPLAIN (ANALYZE, COSTS OFF, SUMMARY OFF, TIMING OFF)
SELECT t1.c3 FROM t1, t2 WHERE t1.c2 = t2.c2;
QUERY PLAN
----------------------------------------------------------------------------------
Gather Motion 1:1 (slice1; segments: 1) (actual rows=32 loops=1)
-> Hash Join (actual rows=32 loops=1)
Hash Cond: (t1.c2 = t2.c2)
Extra Text: Hash chain length 8.0 avg, 8 max, using 4 of 524288 buckets.
-> Seq Scan on t1 (actual rows=131072 loops=1)
-> Hash (actual rows=32 loops=1)
Buckets: 524288 Batches: 1 Memory Usage: 4098kB
-> Seq Scan on t2 (actual rows=32 loops=1)
Optimizer: GPORCA
(9 rows)
Time: 38.471 ms
```
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
Fixes #ISSUE_Number
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2297005813
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
When gp_enable_runtime_filter_pushdown is enabled by default, it applies to all queries on PAX-formatted tables, which is expected to affect TPC-DS/TPC-H performance.
IN and min-max filters rely on column-level statistics, whereas bloom filters perform row-level filtering. Even without IN or min-max filtering, simply pushing the bloom filter from SeqNext() down to the TableAM level can reduce unnecessary memory copies. For example, if a row has already been filtered by a bloom filter expression, there is no need to construct and return the corresponding TableTupleSlot for that row.
It leads to a minor performance improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2314793358
##########
src/backend/executor/nodeSeqscan.c:
##########
@@ -80,13 +80,18 @@ SeqNext(SeqScanState *node)
* node->filter_in_seqscan is false means scankey need to be pushed to
* AM.
*/
- if (gp_enable_runtime_filter_pushdown && !node->filter_in_seqscan)
+ if (gp_enable_runtime_filter_pushdown && node->filters &&
Review Comment:
lost `!node->filter_in_seqscan` condition ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2305856461
##########
contrib/pax_storage/src/cpp/storage/filter/pax_sparse_pg_path.cc:
##########
@@ -57,6 +57,23 @@ void PaxSparseFilter::Initialize(List *quals) {
fl_nodes.emplace_back(std::move(fl_node));
}
+ // walk scan key and only support min/max filter now
+ for (int i = 0; i < nkeys; i++) {
+ // TODO: support bloom filter in PaxFilter
+ // but now just skip it, SeqNext() will check bloom filter in PassByBloomFilter()
+ if (key[i].sk_flags & SK_BLOOM_FILTER) {
+ continue;
+ }
+
+ if (key[i].sk_strategy != BTGreaterEqualStrategyNumber &&
+ key[i].sk_strategy != BTLessEqualStrategyNumber) {
Review Comment:
Can't process column filters with `sk_strategy = BTEqualStrategyNumber`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2296095850
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
Min/Max filters depend on column statistics, but push-down bloom filters can support row-level filtering on the pax table without relying on column statistics. So regardless of whether there is min/max statistics for the column, we can push down the bloom filter
Row-level filtering via bloom filters will be implemented later.
Furthermore, we should consider enabling statistics updates for the PAX table in a bgworker. Currently, minmax_columns needs to be specified when creating a table, and it will slow down the performance of data insertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2296121523
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
that means enable SCAN_SUPPORT_RUNTIME_FILTER default , will impact tpcds, tpch ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2305953554
##########
contrib/pax_storage/src/cpp/storage/filter/pax_sparse_pg_path.cc:
##########
@@ -57,6 +57,23 @@ void PaxSparseFilter::Initialize(List *quals) {
fl_nodes.emplace_back(std::move(fl_node));
}
+ // walk scan key and only support min/max filter now
+ for (int i = 0; i < nkeys; i++) {
+ // TODO: support bloom filter in PaxFilter
+ // but now just skip it, SeqNext() will check bloom filter in PassByBloomFilter()
+ if (key[i].sk_flags & SK_BLOOM_FILTER) {
+ continue;
+ }
+
+ if (key[i].sk_strategy != BTGreaterEqualStrategyNumber &&
+ key[i].sk_strategy != BTLessEqualStrategyNumber) {
Review Comment:
currently the runtime filter will only push down the following three strategies.
https://github.com/apache/cloudberry/blob/768801c3db840250da7d69d1ece13a41a7c35033/src/backend/executor/nodeHash.c#L4169-L4192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2297005813
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
When gp_enable_runtime_filter_pushdown is enabled by default, it applies to all queries on PAX-formatted tables, which is expected to affect TPC-DS/TPC-H performance. IN and min-max filters rely on column-level statistics, whereas bloom filters perform row-level filtering. Even without IN or min-max filtering, simply pushing the bloom filter from SeqNext() down to the TableAM level can reduce unnecessary memory copies. For example, if a row has already been filtered by a bloom filter expression, there is no need to construct and return the corresponding TableTupleSlot for that row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2310135310
##########
contrib/pax_storage/src/cpp/storage/filter/pax_sparse_pg_path.cc:
##########
@@ -57,6 +57,23 @@ void PaxSparseFilter::Initialize(List *quals) {
fl_nodes.emplace_back(std::move(fl_node));
}
+ // walk scan key and only support min/max filter now
+ for (int i = 0; i < nkeys; i++) {
Review Comment:
We add ProcessScanKey nodes after the ExprWalker nodes. Does this mean ExprWalker are executed before ProcessScanKey? If so, maybe it would be cheaper to execute the ProcessScanKey first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2294235481
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
```sql
minmax_columns = cbdb::GetMinMaxColumnIndexes(relation);
if (!minmax_columns.empty()) {
flags |= SCAN_SUPPORT_RUNTIME_FILTER;
}
```
need minmax_columns ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2336274557
##########
contrib/pax_storage/src/cpp/access/pax_scanner.cc:
##########
@@ -326,8 +326,8 @@ void PaxScanDesc::EndScan() {
}
TableScanDesc PaxScanDesc::BeginScanExtractColumns(
- Relation rel, Snapshot snapshot, int /*nkeys*/,
- struct ScanKeyData * /*key*/, ParallelTableScanDesc parallel_scan,
+ Relation rel, Snapshot snapshot, int nkeys,
+ struct ScanKeyData *key, ParallelTableScanDesc parallel_scan,
Review Comment:
It is better to keep the function signature consistent with it in the table access method
```
/*
* GPDB: Extract columns for scan from targetlist and quals. This is mainly
* for AOCS tables.
*/
TableScanDesc (*scan_begin_extractcolumns) (Relation rel,
Snapshot snapshot,
int nkeys, struct ScanKeyData *key,
ParallelTableScanDesc parallel_scan,
struct PlanState *ps,
uint32 flags);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it merged PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2333202158
##########
contrib/pax_storage/src/cpp/access/pax_scanner.cc:
##########
@@ -326,8 +326,8 @@ void PaxScanDesc::EndScan() {
}
TableScanDesc PaxScanDesc::BeginScanExtractColumns(
- Relation rel, Snapshot snapshot, int /*nkeys*/,
- struct ScanKeyData * /*key*/, ParallelTableScanDesc parallel_scan,
+ Relation rel, Snapshot snapshot, int nkeys,
+ struct ScanKeyData *key, ParallelTableScanDesc parallel_scan,
Review Comment:
`struct ScanKeyData *` -> `ScanKey` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "leborchuk (via GitHub)" <gi...@apache.org>.
leborchuk commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2310135310
##########
contrib/pax_storage/src/cpp/storage/filter/pax_sparse_pg_path.cc:
##########
@@ -57,6 +57,23 @@ void PaxSparseFilter::Initialize(List *quals) {
fl_nodes.emplace_back(std::move(fl_node));
}
+ // walk scan key and only support min/max filter now
+ for (int i = 0; i < nkeys; i++) {
Review Comment:
We add ProcessScanKey nodes after the ExprWalker nodes. Does it means ExprWalker nodes are executed before ProcessScanKey? If so, maybe it will be cheaper to execute ProcessScanKey first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2296095850
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
Min/Max filters depend on column statistics, but push-down bloom filters can support row-level filtering on the pax table without relying on column statistics. Row-level filtering via bloom filters will be implemented later.
Furthermore, we should consider enabling statistics updates for the PAX table in a bgworker. Currently, minmax_columns needs to be specified when creating a table, and it will slow down the performance of data insertion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2336274557
##########
contrib/pax_storage/src/cpp/access/pax_scanner.cc:
##########
@@ -326,8 +326,8 @@ void PaxScanDesc::EndScan() {
}
TableScanDesc PaxScanDesc::BeginScanExtractColumns(
- Relation rel, Snapshot snapshot, int /*nkeys*/,
- struct ScanKeyData * /*key*/, ParallelTableScanDesc parallel_scan,
+ Relation rel, Snapshot snapshot, int nkeys,
+ struct ScanKeyData *key, ParallelTableScanDesc parallel_scan,
Review Comment:
It is best to keep the function signature consistent with it in the table access method
```
/*
* GPDB: Extract columns for scan from targetlist and quals. This is mainly
* for AOCS tables.
*/
TableScanDesc (*scan_begin_extractcolumns) (Relation rel,
Snapshot snapshot,
int nkeys, struct ScanKeyData *key,
ParallelTableScanDesc parallel_scan,
struct PlanState *ps,
uint32 flags);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2314767485
##########
contrib/pax_storage/src/cpp/storage/filter/pax_sparse_pg_path.cc:
##########
@@ -57,6 +57,23 @@ void PaxSparseFilter::Initialize(List *quals) {
fl_nodes.emplace_back(std::move(fl_node));
}
+ // walk scan key and only support min/max filter now
+ for (int i = 0; i < nkeys; i++) {
Review Comment:
That makes sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] feat: pushdown the runtime filter to Table AM [cloudberry]
Posted by "gongxun0928 (via GitHub)" <gi...@apache.org>.
gongxun0928 commented on code in PR #1324:
URL: https://github.com/apache/cloudberry/pull/1324#discussion_r2297005813
##########
contrib/pax_storage/src/cpp/access/pax_access_handle.cc:
##########
@@ -453,6 +453,7 @@ uint32 PaxAccessMethod::ScanFlags(Relation relation) {
flags |= SCAN_FORCE_BIG_WRITE_LOCK;
#endif
+ flags |= SCAN_SUPPORT_RUNTIME_FILTER;
Review Comment:
When gp_enable_runtime_filter_pushdown is enabled by default, it applies to all queries on PAX-formatted tables, which is expected to affect TPC-DS/TPC-H performance.
IN and min-max filters rely on column-level statistics, whereas bloom filters perform row-level filtering. Even without IN or min-max filtering, simply pushing the bloom filter from SeqNext() down to the TableAM level can reduce unnecessary memory copies. For example, if a row has already been filtered by a bloom filter expression, there is no need to construct and return the corresponding TableTupleSlot for that row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org