You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "IPetrov2013 (via GitHub)" <gi...@apache.org> on 2026/01/18 03:53:37 UTC
[I] [Bug] cluster crashed [cloudberry]
IPetrov2013 opened a new issue, #1537:
URL: https://github.com/apache/cloudberry/issues/1537
### Apache Cloudberry version
main branch
### What happened
```go
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=11, threadid=140108295803712) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=11, threadid=140108295803712) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140108295803712, signo=signo@entry=11) at ./nptl/pthread_kill.c:89
#3 0x00007f6d83eda476 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#4 0x000055b769bdae1c in StandardHandlerForSigillSigsegvSigbus_OnMainThread (
processName=0x55b76a2eb820 "a startup process", postgres_signal_arg=11) at elog.c:5353
#5 0x000055b7698d00b3 in HandleCrash (postgres_signal_arg=11) at startup.c:198
#6 0x000055b76a1385c5 in wrapper_handler (postgres_signal_arg=11) at pqsignal.c:90
#7 <signal handler called>
#8 0x00000000002f34d0 in ?? ()
#9 0x00007f6d80e75bc3 in PaxXLogDropDatabase (dbid=263719)
at /cloudberry/contrib/pax_storage/src/cpp/storage/wal/paxc_wal.cc:478
#10 0x000055b76939e729 in XLogDropDatabase (dbid=263719) at xlogutils.c:678
#11 0x000055b769547c9d in dbase_redo (record=0x55b770b557b8) at dbcommands.c:2533
#12 0x000055b769387b9e in StartupXLOG () at xlog.c:7885
#13 0x000055b7698d021c in StartupProcessMain () at startup.c:267
#14 0x000055b769400056 in AuxiliaryProcessMain (argc=2, argv=0x7fffd18317e0) at bootstrap.c:484
#15 0x000055b7698ce791 in StartChildProcess (type=StartupProcess) at postmaster.c:6110
#16 0x000055b7698cd11e in PostmasterStateMachine () at postmaster.c:4596
#17 0x000055b7698cb920 in reaper (postgres_signal_arg=17) at postmaster.c:3782
#18 <signal handler called>
#19 0x00007f6d83fb374d in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffd1832820)
at ../sysdeps/unix/sysv/linux/select.c:69
#20 0x000055b76a138145 in pg_usleep (microsec=100000) at pgsleep.c:56
#21 0x000055b7698c8574 in ServerLoop () at postmaster.c:1982
#22 0x000055b7698c7d81 in PostmasterMain (argc=7, argv=0x55b770b014f0) at postmaster.c:1677
#23 0x000055b76972586c in main (argc=7, argv=0x55b770b014f0) at main.c:269
```
### What you think should happen instead
_No response_
### How to reproduce
We have observed intermittent core dumps occurring during gpstop -ar operations on systems containing Pax tables.
### Operating System
all
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] cluster crashed [cloudberry]
Posted by "lss602726449 (via GitHub)" <gi...@apache.org>.
lss602726449 commented on issue #1537:
URL: https://github.com/apache/cloudberry/issues/1537#issuecomment-3767095989
I have try to create pax table or create a database, create pax table, and then drop database. Then I try to restart database by gpstop. But I have not reproduce this problem. Could you please provide more info like sql and operation you do to help me better reproduce this problem. Thanks @IPetrov2013
```
[gpadmin@hashdata cloudberrydb]$ psql postgres -p 7000
psql (14.4, server 14.4)
Type "help" for help.
postgres=# create table test(a int) using pax;
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Apache Cloudberry data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
postgres=# insert into test values(1);
INSERT 0 1
postgres=# select *from test;
a
---
1
(1 row)
postgres=# quit
[gpadmin@hashdata cloudberrydb]$ gpstop ^C
[gpadmin@hashdata cloudberrydb]$ export COORDINATOR_DATA_DIRECTORY=/home/gpadmin/cloudberrydb/gpAux/gpdemo/datadir
s/qddir/demoDataDir-1/
[gpadmin@hashdata cloudberrydb]$ make create-demo-cluster^C
[gpadmin@hashdata cloudberrydb]$ gpstop -ar
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Starting gpstop with args: -ar
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Gathering information and validating the environment...
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Obtaining Cloudberry Coordinator catalog information
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Obtaining Segment details from coordinator...
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Cloudberry Version: 'postgres (Apache Cloudberry) 3.0.0-devel+dev.2219.gc80e93d4d57 build dev'
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Commencing Coordinator instance shutdown with mode='smart'
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Coordinator segment instance directory=/home/gpadmin/cloudberrydb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/
20260119:16:23:10:023206 gpstop:hashdata:gpadmin-[INFO]:-Stopping coordinator segment and waiting for user connections to finish ...
server shutting down
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-Attempting forceful termination of any leftover coordinator process
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-Terminating processes for segment /home/gpadmin/cloudberrydb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-Stopping coordinator standby host hashdata mode=smart
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-Successfully shutdown standby process on hashdata
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-Targeting dbid [2, 5, 3, 6, 4, 7] for shutdown
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20260119:16:23:11:023206 gpstop:hashdata:gpadmin-[INFO]:-0.00% of jobs completed
20260119:16:23:14:023206 gpstop:hashdata:gpadmin-[INFO]:-100.00% of jobs completed
20260119:16:23:14:023206 gpstop:hashdata:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20260119:16:23:14:023206 gpstop:hashdata:gpadmin-[INFO]:-0.00% of jobs completed
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:-100.00% of jobs completed
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:-----------------------------------------------------
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:- Segments stopped successfully = 6
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:- Segments with errors during stop = 0
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:-----------------------------------------------------
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:-Successfully shutdown 6 of 6 segment instances
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20260119:16:23:16:023206 gpstop:hashdata:gpadmin-[INFO]:-Restarting System...
[gpadmin@hashdata cloudberrydb]$ psql postgres -p 7000
psql (14.4, server 14.4)
Type "help" for help.
postgres=# create database test;
CREATE DATABASE
postgres=# \c test;
psql (14.4, server 14.4)
You are now connected to database "test" as user "gpadmin".
test=# create table test(a int) using pax;
NOTICE: Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'a' as the Apache Cloudberry data distribution key for this table.
HINT: The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
test=# insert into test values(1);
INSERT 0 1
test=# \c postgres
psql (14.4, server 14.4)
You are now connected to database "postgres" as user "gpadmin".
postgres=# drop database test;
DROP DATABASE
postgres=# quit
[gpadmin@hashdata cloudberrydb]$ gpstop -ar
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Starting gpstop with args: -ar
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Gathering information and validating the environment...
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Obtaining Cloudberry Coordinator catalog information
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Obtaining Segment details from coordinator...
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Cloudberry Version: 'postgres (Apache Cloudberry) 3.0.0-devel+dev.2219.gc80e93d4d57 build dev'
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Commencing Coordinator instance shutdown with mode='smart'
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Coordinator segment instance directory=/home/gpadmin/cloudberrydb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/
20260119:16:26:48:026180 gpstop:hashdata:gpadmin-[INFO]:-Stopping coordinator segment and waiting for user connections to finish ...
server shutting down
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-Attempting forceful termination of any leftover coordinator process
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-Terminating processes for segment /home/gpadmin/cloudberrydb/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-Stopping coordinator standby host hashdata mode=smart
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-Successfully shutdown standby process on hashdata
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-Targeting dbid [2, 5, 3, 6, 4, 7] for shutdown
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20260119:16:26:49:026180 gpstop:hashdata:gpadmin-[INFO]:-0.00% of jobs completed
20260119:16:26:52:026180 gpstop:hashdata:gpadmin-[INFO]:-100.00% of jobs completed
20260119:16:26:52:026180 gpstop:hashdata:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20260119:16:26:52:026180 gpstop:hashdata:gpadmin-[INFO]:-0.00% of jobs completed
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:-100.00% of jobs completed
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:-----------------------------------------------------
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:- Segments stopped successfully = 6
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:- Segments with errors during stop = 0
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:-----------------------------------------------------
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:-Successfully shutdown 6 of 6 segment instances
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20260119:16:26:54:026180 gpstop:hashdata:gpadmin-[INFO]:-Restarting System...
[gpadmin@hashdata cloudberrydb]$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] cluster crashed [cloudberry]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1537:
URL: https://github.com/apache/cloudberry/issues/1537#issuecomment-3764874981
Hi, @IPetrov2013 welcome!🎊 Thanks for taking the time to point this out.🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] cluster crashed [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on issue #1537:
URL: https://github.com/apache/cloudberry/issues/1537#issuecomment-4140313872
Hi @IPetrov2013,
Thanks for reporting this, and apologies for the delayed follow-up! 🙏
Just wanted to check in — are you still experiencing this crash on recent builds? There have been a number of updates to PAX storage since January, and it's possible this may have been addressed along the way.
If you're able to reproduce it again, any additional details would be very helpful — such as the specific operations performed before the restart (e.g., whether a `DROP DATABASE` containing PAX tables was involved), or the PAX-related WAL records present at the time of the crash. From the stack trace, it looks like the crash occurs in `PaxXLogDropDatabase` during startup WAL replay, which suggests it might be related to replaying a database drop that involved PAX tables.
If this is no longer an issue for you, we'll go ahead and close it. Either way, thanks again for bringing it to our attention! 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] cluster crashed [cloudberry]
Posted by "IPetrov2013 (via GitHub)" <gi...@apache.org>.
IPetrov2013 commented on issue #1537:
URL: https://github.com/apache/cloudberry/issues/1537#issuecomment-3773234607
> I have try to create pax table or create a database, create pax table, and then drop database. Then I try to restart database by gpstop. But I have not reproduce this problem. Could you please provide more info like sql and operation you do to help me better reproduce this problem.
Thanks for checking. We also cannot reproduce it stably. After dropping all PAX tables and switching to AO, the system runs normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] cluster crashed [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on issue #1537:
URL: https://github.com/apache/cloudberry/issues/1537#issuecomment-3766870507
@lss602726449 Could you please take a look, thanks so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] cluster crashed [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it closed issue #1537: [Bug] cluster crashed
URL: https://github.com/apache/cloudberry/issues/1537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org