You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "avamingli (via GitHub)" <gi...@apache.org> on 2025/04/01 04:48:51 UTC
[PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
avamingli opened a new pull request, #1029:
URL: https://github.com/apache/cloudberry/pull/1029
This commit optimizes the materialized view (MV) invalidation mechanism to reduce overhead in OLTP workloads. The AQUMV feature introduced `SetRelativeMatviewAuxStatus()` to invalidate MVs at appropriate times, which GP7 lacks. However, TPC-B testing showed this function adds ~8.72% OLTP overhead due to mandatory metadata operations, even when no MVs are present.
To address this, we added `relmvrefcount` to track the number of MVs dependent on a table. When no MVs reference a table, the system now skips `SetRelativeMatviewAuxStatus()` entirely, avoiding unnecessary catalog access. Benchmark results demonstrate significant improvements: under 20 concurrent connections, transactions increased from 373,511 to 404,976, latency decreased from 16.06ms to 14.813ms, and TPS improved from 1,245.3 to 1,350.2.
This optimization restores performance parity with GP7 in MV-free scenarios while maintaining correctness for tables with MVs, delivering measurable throughput gains and reduced latency in high-concurrency OLTP workloads.
Authored-by: Zhang Mingli avamingli@gmail.com
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
Fixes #ISSUE_Number
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029#issuecomment-2773079772
LGTM, except CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "weinan003 (via GitHub)" <gi...@apache.org>.
weinan003 commented on code in PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029#discussion_r2022512097
##########
src/backend/catalog/gp_matview_aux.c:
##########
@@ -230,6 +232,9 @@ InsertMatviewTablesEntries(Oid mvoid, List *relids)
values[Anum_gp_matview_tables_mvoid - 1] = ObjectIdGetDatum(mvoid);
tup = heap_form_tuple(RelationGetDescr(mtRel), values, nulls);
CatalogTupleInsert(mtRel, tup);
+
+ /* update relation's pg_class entry */
+ addRelationMVRefCount(relid, 1);
Review Comment:
Does it need to invalidate the rel cache after the ref count changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "avamingli (via GitHub)" <gi...@apache.org>.
avamingli commented on code in PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029#discussion_r2024910352
##########
src/backend/catalog/gp_matview_aux.c:
##########
@@ -290,6 +296,10 @@ RemoveMatviewTablesEntries(Oid mvoid)
if (!HeapTupleIsValid(tuple))
continue;
+ /* update relation's pg_class entry */
+ relid = ((Form_gp_matview_tables) GETSTRUCT(tuple))->relid;
Review Comment:
Same approach, no difference. What's your specific concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029#discussion_r2023212652
##########
src/backend/catalog/gp_matview_aux.c:
##########
@@ -290,6 +296,10 @@ RemoveMatviewTablesEntries(Oid mvoid)
if (!HeapTupleIsValid(tuple))
continue;
+ /* update relation's pg_class entry */
+ relid = ((Form_gp_matview_tables) GETSTRUCT(tuple))->relid;
Review Comment:
how to handle partitioned table ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "avamingli (via GitHub)" <gi...@apache.org>.
avamingli commented on code in PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029#discussion_r2024918546
##########
src/backend/catalog/gp_matview_aux.c:
##########
@@ -230,6 +232,9 @@ InsertMatviewTablesEntries(Oid mvoid, List *relids)
values[Anum_gp_matview_tables_mvoid - 1] = ObjectIdGetDatum(mvoid);
tup = heap_form_tuple(RelationGetDescr(mtRel), values, nulls);
CatalogTupleInsert(mtRel, tup);
+
+ /* update relation's pg_class entry */
+ addRelationMVRefCount(relid, 1);
Review Comment:
No need—the function calls CatalogTupleUpdate, which handles relcache(of this rel) invalidation automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029#discussion_r2023212652
##########
src/backend/catalog/gp_matview_aux.c:
##########
@@ -290,6 +296,10 @@ RemoveMatviewTablesEntries(Oid mvoid)
if (!HeapTupleIsValid(tuple))
continue;
+ /* update relation's pg_class entry */
+ relid = ((Form_gp_matview_tables) GETSTRUCT(tuple))->relid;
Review Comment:
how to handle partitioned table and foreign table ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Optimize MV invalidation overhead using reference counting. [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it merged PR #1029:
URL: https://github.com/apache/cloudberry/pull/1029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org