You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "woblerr (via GitHub)" <gi...@apache.org> on 2026/05/03 14:10:21 UTC
[I] [Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted [cloudberry]
woblerr opened a new issue, #1717:
URL: https://github.com/apache/cloudberry/issues/1717
### Apache Cloudberry version
Apache Cloudberry 2.1.0-incubating
### What happened
When running `gpactivatestandby -a -f` after a clean cluster shutdown, the activation fails with a critical error.
`gpactivatestandby` internally calls `gpstart -c` to bring up the standby coordinator in utility mode. However, gpstart itself tries to connect to the catalog as part of its startup sequence. The standby instance starts in recovery mode and connections are rejected immediately with:
```bash
20260503:13:38:37:000773 gpstart:standby:gpadmin-[ERROR]:-gpstart failed. exiting...
Traceback (most recent call last):
File "/usr/local/cloudberry-db/lib/python/gppylib/mainUtils.py", line 365, in simple_main_locked
exitCode = commandObject.run()
File "/usr/local/cloudberry-db/bin/gpstart", line 170, in run
self._startCoordinator()
File "/usr/local/cloudberry-db/bin/gpstart", line 513, in _startCoordinator
self.gparray = GpArray.initFromCatalog(self.dburl, utility=True)
File "/usr/local/cloudberry-db/lib/python/gppylib/gparray.py", line 959, in initFromCatalog
with closing(dbconn.connect(dbURL, utility)) as conn:
File "/usr/local/cloudberry-db/lib/python/gppylib/db/dbconn.py", line 263, in connect
connection = pgdb.connect(**conninfo)
File "/usr/local/cloudberry-db/lib/python/pgdb.py", line 1690, in connect
cnx = _connect(dbname, dbhost, dbport, dbopt, dbuser, dbpasswd)
pg.InternalError: connection to server at "localhost" (::1), port 5432 failed: FATAL: the database system is not accepting connections
DETAIL: Hot standby mode is disabled.
'
stderr=''
20260503:13:38:37:000743 gpactivatestandby:standby:gpadmin-[CRITICAL]:-Error activating standby coordinator: ExecutionError: 'non-zero rc: 2' occurred. Details: 'GPSTART_INTERNAL_COORDINATOR_ONLY=1 && $GPHOME/bin/gpstart -a -c -v -d /data/master/gpseg-1' cmd had rc=2 completed=True halted=False
```
This causes `gpstart` to exit with `rc=2`, which propagates back as a CRITICAL failure in `gpactivatestandby`, aborting the remaining steps — including the cluster restart that would have brought the segments online. The coordinator itself does eventually promote (the trigger file was written before calling gpstart).
It requires an additional cluster restart to correctly start the segments.
### What you think should happen instead
_No response_
### How to reproduce
* Initialize a Cloudberry cluster with a standby coordinator (`gpinitstandby`).
* Perform a clean, graceful shutdown of the cluster (`gpstop -a`).
* On the standby host, run `gpactivatestandby -a -f`.
* Observe the CRITICAL error — activation fails even though the standby coordinator process itself did start.
### Operating System
Ubuntu
### Anything else
_No response_
### Are you willing to submit PR?
- [x] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted [cloudberry]
Posted by "MisterRaindrop (via GitHub)" <gi...@apache.org>.
MisterRaindrop closed issue #1717: [Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted
URL: https://github.com/apache/cloudberry/issues/1717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted [cloudberry]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1717:
URL: https://github.com/apache/cloudberry/issues/1717#issuecomment-4366359715
Hi, @woblerr welcome!🎊 Thanks for taking the time to point this out.🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] gpactivatestandby -f fails with CRITICAL error even though standby is successfully promoted [cloudberry]
Posted by "my-ship-it (via GitHub)" <gi...@apache.org>.
my-ship-it commented on issue #1717:
URL: https://github.com/apache/cloudberry/issues/1717#issuecomment-4403778094
Thanks for the detailed report, @woblerr — the traceback and root-cause walk-through are really helpful here. You've correctly identified the race: `gpactivatestandby` writes the trigger file and then immediately invokes `gpstart -c`, which calls `initFromCatalog` before the standby has finished exiting recovery, so the connection is rejected with *"the database system is not accepting connections / Hot standby mode is disabled"*. The coordinator does promote, but the CRITICAL exit skips the segment-startup step, which is why a second `gpstart` is needed to fully recover.
Waiting for promotion to complete (e.g., polling `pg_is_in_recovery()` or retrying the catalog connection with a bounded timeout) inside `gpactivatestandby` before calling `gpstart -c` looks like the right fix — it keeps the retry logic scoped to the activation path rather than leaking into the general `gpstart` flow.
Looking forward to your PR! Happy to review once it's up. 🙌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org