You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "reshke (via GitHub)" <gi...@apache.org> on 2026/08/18 04:28:13 UTC

[PR] Fix: gpstart waits for coordinator to finish recovery before connecting [cloudberry]

reshke opened a new pull request, #1898:
URL: https://github.com/apache/cloudberry/pull/1898

   In CBDB3 after the upstream PG16 rebase on commit 7ff23c6d
    PMSIGNAL_RECOVERY_STARTED is now sent during crash recovery too.
   
   With hot_standby=off (the default), this causes postmaster to write PM_STATUS_STANDBY to the pidfile as soon as recovery begins, and pg_ctl -w treats standby as success.  gpstart then hits
    FATAL "Hot standby mode is disabled".
   
   Add a shared _waitForCoordinatorRecovery helper that polls dbconn.connect with a 5s interval up to 300s, retrying only on recovery-related FATAL messages (not accepting connections / not yet accepting connections /
   
   
   ```
   reshke@yezzey-cbdb-bench:~/cloudberry$ gpstart -a
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting gpstart with args: -a
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Gathering information and validating the environment...
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Cloudberry Binary Version: 'postgres (Apache Cloudberry) 3.0.0-devel+dev.20675.g51967d33cb2 build dev'
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Cloudberry Catalog Version: '302606111'
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting Coordinator instance in admin mode
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-CoordinatorStart pg_ctl cmd is env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=utility " start
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Obtaining Cloudberry Coordinator catalog information
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Obtaining Segment details from coordinator...
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
   20260818:04:25:06:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Coordinator still recovering, waiting 5 seconds (waited 0/300): connection to server at "localhost" (::1), port 7000 failed: FATAL:  the database system is not accepting connections
   DETAIL:  Hot standby mode is disabled.
   
   20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
   20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
   20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Setting new coordinator era
   20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Coordinator Started...
   20260818:04:25:11:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Shutting down coordinator
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Process results...
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-----------------------------------------------------
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-   Successful segment starts                                            = 6
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-   Failed segment starts                                                = 0
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-----------------------------------------------------
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Successfully started 6 of 6 segment instances
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-----------------------------------------------------
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting Coordinator instance yezzey-cbdb-bench directory /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-CoordinatorStart pg_ctl cmd is env GPSESSID=0000000000 GPERA=850d57def3e5611f_260818042511 $GPHOME/bin/pg_ctl -D /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1 -l /home/reshke/cloudberry/gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/startup.log -w -t 600 -o " -p 7000 -c gp_role=dispatch " start
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Command pg_ctl reports Coordinator yezzey-cbdb-bench instance active
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Connecting to db template1 on host localhost
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Starting standby coordinator
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Checking if standby coordinator is running on host: yezzey-cbdb-bench  in directory: /home/reshke/cloudberry/gpAux/gpdemo/datadirs/standby
   20260818:04:25:12:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-CoordinatorStart pg_ctl cmd is env GPSESSID=0000000000 GPERA=850d57def3e5611f_260818042511 $GPHOME/bin/pg_ctl -D /home/reshke/cloudberry/gpAux/gpdemo/datadirs/standby -l /home/reshke/cloudberry/gpAux/gpdemo/datadirs/standby/log/startup.log -t 600 -o " -p 7001 -c gp_role=dispatch " start
   20260818:04:25:13:1743452 gpstart:yezzey-cbdb-bench:reshke-[INFO]:-Database successfully started
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Wait for coordinator to finish recovery before connecting in gpstart [cloudberry]

Posted by "reshke (via GitHub)" <gi...@apache.org>.
reshke commented on PR #1898:
URL: https://github.com/apache/cloudberry/pull/1898#issuecomment-5324027203

   To trigger issue you need to dirty your buffers and run immediate stop:
   
   ```
   CREATE TABLE z(i int); INSERT INTO z SELECT generate_series(1, 1000000); 
   ```
   and then gpstop -a -i && gpstart -a
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Wait for coordinator to finish recovery before connecting in gpstart [cloudberry]

Posted by "reshke (via GitHub)" <gi...@apache.org>.
reshke commented on PR #1898:
URL: https://github.com/apache/cloudberry/pull/1898#issuecomment-5331489945

   Another option is to change pg_ctl wait semantic for CBDB. But I consider this as undesirable discrepancy with upstream.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Wait for coordinator to finish recovery before connecting in gpstart [cloudberry]

Posted by "reshke (via GitHub)" <gi...@apache.org>.
reshke merged PR #1898:
URL: https://github.com/apache/cloudberry/pull/1898


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org