You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "Smyatkin-Maxim (via GitHub)" <gi...@apache.org> on 2025/04/29 07:42:03 UTC

[PR] Fix segfault when recovering 2pc transaction [cloudberry]

Smyatkin-Maxim opened a new pull request, #1078:
URL: https://github.com/apache/cloudberry/pull/1078

   When promoting a mirror segment due to failover we have seen a stacktrace like this:
   
   ```
   "FATAL","58P01","requested WAL segment pg_xlog/00000023000071D50000001F has already been removed",,,,,,,0,,"xlogutils.c",580,"Stack
   trace:
   1    0x557bdb9f09b6 postgres errstart + 0x236
   2    0x557bdb5fc6cf postgres <symbol not found> + 0xdb5fc6cf
   3    0x557bdb5fd021 postgres read_local_xlog_page + 0x191
   4    0x557bdb5fb922 postgres <symbol not found> + 0xdb5fb922
   5    0x557bdb5fba11 postgres XLogReadRecord + 0xa1
   6    0x557bdb5e7767 postgres RecoverPreparedTransactions + 0xd7
   7    0x557bdb5f608b postgres StartupXLOG + 0x2a3b
   8    0x557bdb870a89 postgres StartupProcessMain + 0x139
   9    0x557bdb62f489 postgres AuxiliaryProcessMain + 0x549
   10   0x557bdb86d275 postgres <symbol not found> + 0xdb86d275
   11   0x557bdb8704e3 postgres PostmasterMain + 0x1213
   12   0x557bdb56a1f7 postgres main + 0x497
   13   0x7fded4a61c87 libc.so.6 __libc_start_main + 0xe7
   ```
   Note: stacktrace is from one of our production GP clusters and might be slightly different from what we will see in cloudberry, but the failure is still present here as well. Testcase proves it.
   
   PG13 and PG14 have a fix for this bug, but it's doesn't have any test case and looks like we didn't cherry-pick that far. The discussion can be found here: https://www.postgresql.org/message-id/flat/743b9b45a2d4013bd90b6a5cba8d6faeb717ee34.camel%40cybertec.at
   
   In a few words, StartupXLOG() renames the last wal segment to .partial but tries to read it by the old name later in RecoverPreparedTransactions().
   
   The fix is mostly borrowed from PG14 postgres/postgres@f663b00 with some cloudberry-related exceptions. Also added a regression test which segfaults without this fix for any version of GP, PG<=12 or Cloudberry.
   
   <!-- Thank you for your contribution to Apache Cloudberry (Incubating)! -->
   
   Fixes #ISSUE_Number
   
   ### What does this PR do?
   <!-- Brief overview of the changes, including any major features or fixes -->
   
   ### Type of Change
   - [ ] Bug fix (non-breaking change)
   - [ ] New feature (non-breaking change)
   - [ ] Breaking change (fix or feature with breaking changes)
   - [ ] Documentation update
   
   ### Breaking Changes
   <!-- Remove if not applicable. If yes, explain impact and migration path -->
   
   ### Test Plan
   <!-- How did you test these changes? -->
   - [ ] Unit tests added/updated
   - [ ] Integration tests added/updated
   - [ ] Passed `make installcheck`
   - [ ] Passed `make -C src/test installcheck-cbdb-parallel`
   
   ### Impact
   <!-- Remove sections that don't apply -->
   **Performance:**
   <!-- Any performance implications? -->
   
   **User-facing changes:**
   <!-- Any changes visible to users? -->
   
   **Dependencies:**
   <!-- New dependencies or version changes? -->
   
   ### Checklist
   - [ ] Followed [contribution guide](https://cloudberry.apache.org/contribute/code)
   - [ ] Added/updated documentation
   - [ ] Reviewed code for security implications
   - [ ] Requested review from [cloudberry committers](https://github.com/orgs/apache/teams/cloudberry-committers)
   
   ### Additional Context
   <!-- Any other information that would help reviewers? Remove if none -->
   
   ### CI Skip Instructions
   <!--
   To skip CI builds, add the appropriate CI skip identifier to your PR title.
   The identifier must:
   - Be in square brackets []
   - Include the word "ci" and either "skip" or "no"
   - Only use for documentation-only changes or when absolutely necessary
   -->
   
   ---
   <!-- Join our community:
   - Mailing list: [dev@cloudberry.apache.org](https://lists.apache.org/list.html?dev@cloudberry.apache.org) (subscribe: dev-subscribe@cloudberry.apache.org)
   - Discussions: https://github.com/apache/cloudberry/discussions -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "Smyatkin-Maxim (via GitHub)" <gi...@apache.org>.
Smyatkin-Maxim commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2838122549

   I see that my test is failing. Didn't fail locally the last time I tried. Will take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2842969436

   @jiaqizho @gongxun0928 
   ```sql
   1 | lock_tbl1a
                 1 | lock_view2
                 1 | lock_view3
   +             1 | pg_pax_blocks_4[33](https://github.com/apache/cloudberry/actions/runs/14729599874/job/41419204900?pr=1078#step:18:34)56
                 2 | lock_tbl1
                 2 | lock_tbl1a
                 2 | lock_view2
   ```
   this pax related.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2838988038

   Nice catch! @Smyatkin-Maxim  it's pax related flake test. So I will retry ci.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "jiaqizho (via GitHub)" <gi...@apache.org>.
jiaqizho commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2896404106

   > @jiaqizho @gongxun0928
   > 
   > ```sql
   > 1 | lock_tbl1a
   >               1 | lock_view2
   >               1 | lock_view3
   > +             1 | pg_pax_blocks_4[33](https://github.com/apache/cloudberry/actions/runs/14729599874/job/41419204900?pr=1078#step:18:34)56
   >               2 | lock_tbl1
   >               2 | lock_tbl1a
   >               2 | lock_view2
   > ```
   > 
   > this pax related.
   
   retry pls. not sure which case will cause this lock.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2942412520

   > @jiaqizho @gongxun0928
   > 
   > ```sql
   > 1 | lock_tbl1a
   >               1 | lock_view2
   >               1 | lock_view3
   > +             1 | pg_pax_blocks_4[33](https://github.com/apache/cloudberry/actions/runs/14729599874/job/41419204900?pr=1078#step:18:34)56
   >               2 | lock_tbl1
   >               2 | lock_tbl1a
   >               2 | lock_view2
   > ```
   > 
   > this pax related.
   
   The test should be fixed in the latest main branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "Smyatkin-Maxim (via GitHub)" <gi...@apache.org>.
Smyatkin-Maxim commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2841310063

   > Nice catch! @Smyatkin-Maxim it's pax related flake test. So I will retry ci.
   
   on the first CI run there was a crash+coredump on my testcase. I'm gonna be rerunning IC2 tests a couple times, feels like there is yet another hardly reproducible bug, which my testcase (or my fix?) uncovers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "Smyatkin-Maxim (via GitHub)" <gi...@apache.org>.
Smyatkin-Maxim commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2837825880

   Hi team,
   as this PR somewhat reorders the process of xlog startup, I need reviewers to carefully double-check I didn't mess something up here. Especially I have no knowledge how Startup hook is used, perhaps it cannot be moved up this way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "yjhjstz (via GitHub)" <gi...@apache.org>.
yjhjstz merged PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix segfault when recovering 2pc transaction [cloudberry]

Posted by "gfphoenix78 (via GitHub)" <gi...@apache.org>.
gfphoenix78 commented on PR #1078:
URL: https://github.com/apache/cloudberry/pull/1078#issuecomment-2947707115

   Thanks for your PR, @Smyatkin-Maxim, do you mind to cherry-pick the [commit](https://github.com/postgres/postgres/commit/f663b009189ff175a8d134a6cc6e26b5940a1449) that fixes this issue in pg14? So, the context and commit message are kept. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org