You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "fanfuxiaoran (via GitHub)" <gi...@apache.org> on 2025/06/10 10:05:12 UTC
[PR] [WIP]:Cherry pick hot standby commits from gpdb [cloudberry]
fanfuxiaoran opened a new pull request, #1152:
URL: https://github.com/apache/cloudberry/pull/1152
<!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
Fixes #ISSUE_Number
### What does this PR do?
<!-- Brief overview of the changes, including any major features or fixes -->
### Type of Change
- [ ] Bug fix (non-breaking change)
- [ ] New feature (non-breaking change)
- [ ] Breaking change (fix or feature with breaking changes)
- [ ] Documentation update
### Breaking Changes
<!-- Remove if not applicable. If yes, explain impact and migration path -->
### Test Plan
<!-- How did you test these changes? -->
- [ ] Unit tests added/updated
- [ ] Integration tests added/updated
- [ ] Passed `make installcheck`
- [ ] Passed `make -C src/test installcheck-cbdb-parallel`
### Impact
<!-- Remove sections that don't apply -->
**Performance:**
<!-- Any performance implications? -->
**User-facing changes:**
<!-- Any changes visible to users? -->
**Dependencies:**
<!-- New dependencies or version changes? -->
### Checklist
- [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
- [ ] Added/updated documentation
- [ ] Reviewed code for security implications
- [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
### Additional Context
<!-- Any other information that would help reviewers? Remove if none -->
### CI Skip Instructions
<!--
To skip CI builds, add the appropriate CI skip identifier to your PR title.
The identifier must:
- Be in square brackets []
- Include the word "ci" and either "skip" or "no"
- Only use for documentation-only changes or when absolutely necessary
-->
---
<!-- Join our community:
- Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
- Discussions: https://github.com/apache/cloudberry/discussions -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by Xiaoran Wang <fa...@gmail.com>.
Actually no. EnableHotStandby only will be true on hot_standby cluster. On
the primary cluster,
it's always false. But the distrib_xid is written to xlog by primary
cluster.
Discussed with Max. We prefer to use a Var , may be named
'enable_distri_xid' , it is like the var '
NEXTGXID', it will be both store in a configure file and xlog. Then it will
be synced to standby
cluster.
The standby cluster will check the variable 'enable_distri_xid', if it's
true, then the cluster
cannot be configured as hot_standby.
How do you think about this idea?
yjhjstz (via GitHub) <gi...@apache.org> 于2025年6月18日周三 23:22写道：
>
> yjhjstz commented on code in PR #1152:
> URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2154899650
>
>
> ##########
> src/backend/access/transam/xact.c:
> ##########
> @@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
> xl_xact_distrib xl_distrib;
> xl_xact_deldbs xl_deldbs;
> XLogRecPtr recptr;
> - bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE &&
> MyTmGxactLocal->isOnePhaseCommit);
> bool isDtxPrepared = isPreparedDtxTransaction();
> + DistributedTransactionId distrib_xid =
> getDistributedTransactionId();
>
> Review Comment:
> can we use `EnableHotStandby` to control distrib_xid usage ?
>
>
>
> --
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
> For additional commands, e-mail: commits-help@cloudberry.apache.org
>
>
--
Best regards !
Xiaoran Wang
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "fanfuxiaoran (via GitHub)" <gi...@apache.org>.
fanfuxiaoran commented on code in PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2157892604
##########
src/backend/access/transam/xact.c:
##########
@@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_distrib xl_distrib;
xl_xact_deldbs xl_deldbs;
XLogRecPtr recptr;
- bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE && MyTmGxactLocal->isOnePhaseCommit);
bool isDtxPrepared = isPreparedDtxTransaction();
+ DistributedTransactionId distrib_xid = getDistributedTransactionId();
Review Comment:
Yes. I'm thinking about to add a job to run installcheck-hot-standby in pipeline? @edespino Could you help to add it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-3149881027
Close the PR since another PR: https://github.com/apache/cloudberry/pull/1268 filed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it closed pull request #1152: Cherry pick hot standby commits from gpdb
URL: https://github.com/apache/cloudberry/pull/1152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2961488589
> > > due to increased size of wal, that make tpcc performance degradation.
> >
> >
> > Yes, It will indeed increase the size of the wal-log. Does it impact performance due to increasing the io of flushing wal-log? or the race of writing the wal-log?
>
> Based on previous tests, adding dxid information has a relatively large impact on TPCC because of wal size. However, to support HOT Standby, we need to add this information, which is a contradiction.
Maybe it's possible to add a GUC to control the behavior, but it increases the complexity of understanding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "fanfuxiaoran (via GitHub)" <gi...@apache.org>.
fanfuxiaoran commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2961355559
> due to increased size of wal, that make tpcc performance degradation.
Yes, It will indeed increase the size of the wal-log. Does it impact performance due to increasing the io of flushing wal-log? or the race of writing the wal-log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2157256759
##########
src/backend/access/transam/xact.c:
##########
@@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_distrib xl_distrib;
xl_xact_deldbs xl_deldbs;
XLogRecPtr recptr;
- bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE && MyTmGxactLocal->isOnePhaseCommit);
bool isDtxPrepared = isPreparedDtxTransaction();
+ DistributedTransactionId distrib_xid = getDistributedTransactionId();
Review Comment:
maybe we can optimize wal to counteract this influence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2157562502
##########
src/backend/access/transam/xact.c:
##########
@@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_distrib xl_distrib;
xl_xact_deldbs xl_deldbs;
XLogRecPtr recptr;
- bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE && MyTmGxactLocal->isOnePhaseCommit);
bool isDtxPrepared = isPreparedDtxTransaction();
+ DistributedTransactionId distrib_xid = getDistributedTransactionId();
Review Comment:
we should reslove conflict then add installcheck-hot-standby pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2158968532
##########
src/backend/access/transam/xact.c:
##########
@@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_distrib xl_distrib;
xl_xact_deldbs xl_deldbs;
XLogRecPtr recptr;
- bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE && MyTmGxactLocal->isOnePhaseCommit);
bool isDtxPrepared = isPreparedDtxTransaction();
+ DistributedTransactionId distrib_xid = getDistributedTransactionId();
Review Comment:
add like below in `.github/workflows/build-cloudberry.yml`
```sql
{"test":"ic-isolation2",
"make_configs":["src/test/isolation2:installcheck-isolation2"]
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2961119298
[Revert "Include distributed xid in transaction commit WAL in all cases"](https://github.com/apache/cloudberry/pull/1152/commits/72cf24513deca8be0743c37906e289b921450b6a)
this pr affact tpcc .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "fanfuxiaoran (via GitHub)" <gi...@apache.org>.
fanfuxiaoran commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2961159185
> [Revert "Include distributed xid in transaction commit WAL in all cases"](https://github.com/apache/cloudberry/pull/1152/commits/72cf24513deca8be0743c37906e289b921450b6a)
>
> this pr affact tpcc .
Hmm... Could you explain more about it ? @yjhjstz The hot standby needs this commit to build distributed snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "fanfuxiaoran (via GitHub)" <gi...@apache.org>.
fanfuxiaoran commented on code in PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2156493279
##########
src/backend/access/transam/xact.c:
##########
@@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_distrib xl_distrib;
xl_xact_deldbs xl_deldbs;
XLogRecPtr recptr;
- bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE && MyTmGxactLocal->isOnePhaseCommit);
bool isDtxPrepared = isPreparedDtxTransaction();
+ DistributedTransactionId distrib_xid = getDistributedTransactionId();
Review Comment:
Actually no. EnableHotStandby only will be true on hot_standby cluster. On the primary cluster,
it's always false. But the distrib_xid is written to xlog by primary cluster.
Discussed with Max. We prefer to use a Var , may be named 'enable_distri_xid' , it is like the var '
NEXTGXID', it will be both store in a configure file and xlog. Then it will be synced to standby
cluster.
The standby cluster will check the variable 'enable_distri_xid', if it's true, then the cluster
cannot be configured as hot_standby.
How do you think about this idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2982627340
after merge https://github.com/apache/cloudberry/pull/1078, there is conflicts at `src/backend/access/transam/xlog.c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2961476062
> > due to increased size of wal, that make tpcc performance degradation.
>
> Yes, It will indeed increase the size of the wal-log. Does it impact performance due to increasing the io of flushing wal-log? or the race of writing the wal-log?
Based on previous tests, adding dxid information has a relatively large impact on TPCC. However, to support HOT Standby, we need to add this information, which is a contradiction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2961178240
due to increased size of wal, that make tpcc performance degradation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "fanfuxiaoran (via GitHub)" <gi...@apache.org>.
fanfuxiaoran commented on PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#issuecomment-2982793995
> after merge #1078, there is conflicts at `src/backend/access/transam/xlog.c`.
Thanks for pointing it out. I will take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [PR] Cherry pick hot standby commits from gpdb [cloudberry]
Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on code in PR #1152:
URL: https://github.com/apache/cloudberry/pull/1152#discussion_r2154899650
##########
src/backend/access/transam/xact.c:
##########
@@ -6880,8 +6884,8 @@ XactLogCommitRecord(TimestampTz commit_time,
xl_xact_distrib xl_distrib;
xl_xact_deldbs xl_deldbs;
XLogRecPtr recptr;
- bool isOnePhaseQE = (Gp_role == GP_ROLE_EXECUTE && MyTmGxactLocal->isOnePhaseCommit);
bool isDtxPrepared = isPreparedDtxTransaction();
+ DistributedTransactionId distrib_xid = getDistributedTransactionId();
Review Comment:
can we use `EnableHotStandby` to control distrib_xid usage ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org