You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "vsbace (via GitHub)" <gi...@apache.org> on 2025/10/17 09:22:18 UTC
[I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
vsbace opened a new issue, #1404:
URL: https://github.com/apache/cloudberry/issues/1404
### Apache Cloudberry version
[apache-cloudberry-2.0.0-incubating]
(https://github.com/apache/cloudberry/releases/tag/2.0.0-incubating)
### What happened
By network problem some primary segments can not be avilable and switched to FAILED state.
When network problem was sovled i can try to recover failed segment and happen the next problem:
`
Continue with segment recovery procedure Yy|Nn (default=N):
> y
20251017:10:58:19:792595 gprecoverseg:cbdb01:gpadmin-[INFO]:-Starting to create new pg_hba.conf on primary segments
20251017:10:58:20:792595 gprecoverseg:cbdb01:gpadmin-[INFO]:-Successfully modified pg_hba.conf on primary segments to allow replication connections
20251017:10:58:20:792595 gprecoverseg:cbdb01:gpadmin-[INFO]:-1 segment(s) to recover
20251017:10:58:20:792595 gprecoverseg:cbdb01:gpadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped
**20251017:10:58:21:792595 gprecoverseg:cbdb01:gpadmin-[WARNING]:-Unable to determine if /u00/cbdb/primary/gpseg7 is symlink. Assuming it is not symlink**
**20251017:10:58:21:792595 gprecoverseg:cbdb01:gpadmin-[CRITICAL]:-gprecoverseg failed. (Reason='invalid literal for int() with base 10:** "# --------------------------------------------------------------------\n# NOTICE from the Apache Cloudberry PPMC\n# --------------------------------------------------------------------\n# This file u') exiting...
`
Yarly i already happened this problem in Cloudberry 1.6 and found WA,
first i need to delete data folder for segment and recover procedure processed successfully.
But i think when i have 15 primary segments i can use WA, but on PRod we have more that segments and it will be a big problem.
### What you think should happen instead
_No response_
### How to reproduce
- one or more primary segments was failed and switched to mirror segmets
- we can try to recover segments by command "GPRECOVERSEG" or "GPRECOVERSEG -F" and catch the same problem
### Operating System
Debian 12.0
### Anything else
This problem occured every time, every run gprecoverseg.
### Are you willing to submit PR?
- [ ] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "vsbace (via GitHub)" <gi...@apache.org>.
vsbace commented on issue #1404:
URL: https://github.com/apache/cloudberry/issues/1404#issuecomment-3414847696
5 nodes in CB Cluster
1 node - Master and Segment node
2-4 nodes - only Segment Nodes
15 primary segments total in cluster
15 mirror segments total in cluster
3 primaries per Node
3 mirrories per Node
select * from gp_segment_configuration
order by 1,2
dbid | content | role | preferred_role | mode | status | port | hostname | address | datadir | warehouseid
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
1 | -1 | p | p | n | u | 5432 | cbdb01 | cbdb01 | /u00/cbdb/master-1/gpseg-1 | 0
2 | 0 | p | p | s | u | 20000 | cbdb01 | cbdb01 | /u00/cbdb/primary/gpseg0 | 0
3 | 1 | p | p | s | u | 20001 | cbdb01 | cbdb01 | /u00/cbdb/primary/gpseg1 | 0
4 | 2 | p | p | s | u | 20002 | cbdb01 | cbdb01 | /u00/cbdb/primary/gpseg2 | 0
5 | 3 | p | p | s | u | 20000 | cbdb02 | cbdb02 | /u00/cbdb/primary/gpseg3 | 0
6 | 4 | p | p | s | u | 20001 | cbdb02 | cbdb02 | /u00/cbdb/primary/gpseg4 | 0
7 | 5 | p | p | s | u | 20002 | cbdb02 | cbdb02 | /u00/cbdb/primary/gpseg5 | 0
8 | 6 | p | p | s | u | 20000 | cbdb03 | cbdb03 | /u00/cbdb/primary/gpseg6 | 0
9 | 7 | p | p | s | u | 20001 | cbdb03 | cbdb03 | /u00/cbdb/primary/gpseg7 | 0
10 | 8 | p | p | s | u | 20002 | cbdb03 | cbdb03 | /u00/cbdb/primary/gpseg8 | 0
11 | 9 | p | p | s | u | 20000 | cbdb04 | cbdb04 | /u00/cbdb/primary/gpseg9 | 0
12 | 10 | p | p | s | u | 20001 | cbdb04 | cbdb04 | /u00/cbdb/primary/gpseg10 | 0
13 | 11 | p | p | s | u | 20002 | cbdb04 | cbdb04 | /u00/cbdb/primary/gpseg11 | 0
14 | 12 | p | p | s | u | 20000 | cbdb05 | cbdb05 | /u00/cbdb/primary/gpseg12 | 0
15 | 13 | p | p | s | u | 20001 | cbdb05 | cbdb05 | /u00/cbdb/primary/gpseg13 | 0
16 | 14 | p | p | s | u | 20002 | cbdb05 | cbdb05 | /u00/cbdb/primary/gpseg14 | 0
17 | 0 | m | m | s | u | 30000 | cbdb02 | cbdb02 | /u00/cbdb/mirror/gpseg0 | 0
18 | 1 | m | m | s | u | 30001 | cbdb02 | cbdb02 | /u00/cbdb/mirror/gpseg1 | 0
19 | 2 | m | m | s | u | 30002 | cbdb02 | cbdb02 | /u00/cbdb/mirror/gpseg2 | 0
20 | 3 | m | m | s | u | 30000 | cbdb03 | cbdb03 | /u00/cbdb/mirror/gpseg3 | 0
21 | 4 | m | m | s | u | 30001 | cbdb03 | cbdb03 | /u00/cbdb/mirror/gpseg4 | 0
22 | 5 | m | m | s | u | 30002 | cbdb03 | cbdb03 | /u00/cbdb/mirror/gpseg5 | 0
23 | 6 | m | m | s | u | 30000 | cbdb04 | cbdb04 | /u00/cbdb/mirror/gpseg6 | 0
24 | 7 | m | m | s | u | 30001 | cbdb04 | cbdb04 | /u00/cbdb/mirror/gpseg7 | 0
25 | 8 | m | m | s | u | 30002 | cbdb04 | cbdb04 | /u00/cbdb/mirror/gpseg8 | 0
26 | 9 | m | m | s | u | 30000 | cbdb05 | cbdb05 | /u00/cbdb/mirror/gpseg9 | 0
27 | 10 | m | m | s | u | 30001 | cbdb05 | cbdb05 | /u00/cbdb/mirror/gpseg10 | 0
28 | 11 | m | m | s | u | 30002 | cbdb05 | cbdb05 | /u00/cbdb/mirror/gpseg11 | 0
29 | 12 | m | m | s | u | 30000 | cbdb01 | cbdb01 | /u00/cbdb/mirror/gpseg12 | 0
30 | 13 | m | m | s | u | 30001 | cbdb01 | cbdb01 | /u00/cbdb/mirror/gpseg13 | 0
31 | 14 | m | m | s | u | 30002 | cbdb01 | cbdb01 | /u00/cbdb/mirror/gpseg14 | 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "vsbace (via GitHub)" <gi...@apache.org>.
vsbace commented on issue #1404:
URL: https://github.com/apache/cloudberry/issues/1404#issuecomment-3422618977
@edespino, Hello, i tryed presented WA and all tests was successfully.
Thank you very much for faster solving issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "edespino (via GitHub)" <gi...@apache.org>.
edespino closed issue #1404: [Bug] GPRECOVERSEG does not work as required
URL: https://github.com/apache/cloudberry/issues/1404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "edespino (via GitHub)" <gi...@apache.org>.
edespino commented on issue #1404:
URL: https://github.com/apache/cloudberry/issues/1404#issuecomment-3416272508
@vsbace Cool ... let me see if I can reproduce this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "edespino (via GitHub)" <gi...@apache.org>.
edespino commented on issue #1404:
URL: https://github.com/apache/cloudberry/issues/1404#issuecomment-3419282843
## Investigation Summary: gprecoverseg PPMC Notice Parsing Error
### Bug Report
User reported `gprecoverseg` failure with the following error:
20251017:10:58:21:792595 gprecoverseg:cbdb01:gpadmin-[CRITICAL]:-gprecoverseg failed.
(Reason='invalid literal for int() with base 10: "# --------------------------------------------------------------------\n
NOTICE from the Apache Cloudberry PPMC\n# --------------------------------------------------------------------\n
This file u') exiting...
The error indicates that `gprecoverseg` attempted to parse a string containing the Apache Cloudberry PPMC notice header as an integer, causing a Python `ValueError`.
---
### Investigation Process
We conducted systematic testing to identify the root cause:
#### ✅ Test 1: SSH Command Output (CLEAN)
```bash
ssh sdw1 "echo 12345"
# Output: 12345 (no contamination)
ssh -t sdw1 "echo 12345"
# Output: 12345 (no contamination)
Result: SSH is NOT the contamination vector. The .bashrc interactive check prevents greenplum_path.sh from executing during non-interactive SSH sessions.
✅ Test 2: Configuration Files in Segment Data Directories (CLEAN)
for host in cdw sdw1 sdw2 sdw3 sdw4; do
ssh $host "find /data* -type f \( -name '*.conf' -o -name '*.opts' \) | \
xargs grep -l 'NOTICE from the Apache Cloudberry PPMC' 2>/dev/null"
done
Result: No contaminated files found in segment data directories on a clean installation.
📋 Analysis of .bashrc Protection
All segment hosts have this protection in .bashrc:
# If not running interactively, don't do anything
case $- in
*i*) ;;
*) return;;
esac
# ... later ...
if [ -f /usr/local/cloudberry/greenplum_path.sh ]; then
source /usr/local/cloudberry/greenplum_path.sh
fi
This prevents greenplum_path.sh from executing during non-interactive SSH, which is why the bug cannot be reproduced on clean systems.
---
Root Cause Analysis
Based on the error message and investigation, the bug occurs when:
1. gprecoverseg reads a file that should contain numeric data (PID, port, dbid, etc.)
2. That file has been contaminated with the PPMC notice header text
3. Python attempts int(file_content) and raises ValueError
The contamination likely occurred during one of these scenarios:
Scenario A: Version Upgrade with Script Output Redirection
# During cluster upgrade or reconfiguration
source /usr/local/cloudberry/greenplum_path.sh # PPMC notice prints to stdout
some_command_that_generates_config > /data/primary/gpseg7/some_file
# PPMC notice gets written into the file
Scenario B: Custom Wrapper Scripts
A wrapper script that sources greenplum_path.sh with stdout redirected:
#!/bin/bash
source /usr/local/cloudberry/greenplum_path.sh > recovery_config.txt
# PPMC notice becomes first lines of recovery_config.txt
Scenario C: Earlier greenplum_path.sh Versions
Earlier versions of greenplum_path.sh may have:
- Printed to stdout unconditionally (not checking for interactive shells)
- Printed before the interactive check in certain initialization scenarios
- Been sourced from /etc/profile or /etc/bash.bashrc (runs before .bashrc protection)
---
Why This Bug is Hard to Reproduce
On clean, newly-initialized clusters:
- ✅ .bashrc interactive check prevents PPMC notice during SSH
- ✅ No contaminated files exist in data directories
- ✅ greenplum_path.sh notice is properly isolated from command output
The bug only affects systems where:
- ❌ Cluster was initialized/upgraded during a transition period with a different greenplum_path.sh behavior
- ❌ Custom scripts source greenplum_path.sh with redirected stdout
- ❌ Files were written during an interactive session that captured the notice
---
Potential Files That Could Be Contaminated
Based on the error context (/u00/cbdb/primary/gpseg7), candidates include:
1. Recovery metadata files - gprecoverseg-specific configuration
2. postmaster.opts - May be parsed for port numbers
3. postgresql.auto.conf - Auto-generated configuration
4. internal.auto.conf - Internal Cloudberry settings
5. Custom recovery tracking files - Created during previous recovery attempts
---
Recommendations
For Users Experiencing This Bug
1. Identify contaminated files:
# On affected segment host
grep -r "NOTICE from the Apache Cloudberry PPMC" /u00/cbdb/primary/gpseg7/
2. Clean the contaminated files:
- Remove the PPMC notice header lines
- Restore from backup if available
- Reinitialize the segment if necessary
3. Prevent recurrence:
- Audit custom scripts that source greenplum_path.sh
- Ensure stdout redirection doesn't capture notice text
- Check /etc/profile, /etc/bash.bashrc for unsafe sourcing
For Cloudberry Development Team
1. Fix greenplum_path.sh to NEVER print to stdout:
# Print notice to stderr instead
echo "NOTICE..." >&2
# OR: Only show in truly interactive shells
if [[ $- == *i* ]] && [[ -t 0 ]]; then
echo "NOTICE..."
fi
2. Add input validation in gprecoverseg:
# Before: int(file_content)
# After:
content = file_content.strip()
if not content.isdigit():
raise ValueError(f"Expected numeric value, got: {content[:50]}")
3. Add file content checks during recovery:
- Detect unexpected content in configuration files
- Provide clear error messages about contamination
- Auto-clean known contamination patterns
4. Document the issue:
- Add to upgrade notes
- Include in troubleshooting guide
- Warn about stdout redirection with greenplum_path.sh
---
Verification Needed
To fully confirm the root cause, we need:
1. Access to the affected system to examine contaminated files
2. gprecoverseg source code review to identify exactly which file it's parsing
3. Version history of when the PPMC notice was added to greenplum_path.sh
---
Related Issues
This is similar to issues seen in other database systems where:
- Shell scripts print banners/notices to stdout
- Tools parse command output expecting clean numeric values
- SSH wrappers or automation scripts get contaminated output
Best Practice: Shell scripts that are sourced should NEVER print to stdout unless that's their explicit purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "edespino (via GitHub)" <gi...@apache.org>.
edespino commented on issue #1404:
URL: https://github.com/apache/cloudberry/issues/1404#issuecomment-3423047289
@vsbace I'm glad it helped. I will close this issue now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org
Re: [I] [Bug] GPRECOVERSEG does not work as required [cloudberry]
Posted by "edespino (via GitHub)" <gi...@apache.org>.
edespino commented on issue #1404:
URL: https://github.com/apache/cloudberry/issues/1404#issuecomment-3414621832
@vsbace Can you share the cluster topology (number of nodes, number of primaries and mirrors per node)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org