[I] [Bug] Mirrors crash during WAL replay when wal_compression is pglz/lz4 [cloudberry]

["roseduan (via GitHub)" <gi...@apache.org>]

roseduan opened a new issue, #2137:
URL: https://github.com/apache/cloudberry/issues/2137

   ### Apache Cloudberry version
   
   _No response_
   
   ### What happened
   
     Reproduce (on a cluster with mirrors):
   ```sql
     CREATE TABLE w (a int, b text) DISTRIBUTED BY (a);
     INSERT INTO w SELECT i, repeat('x', 100) FROM generate_series(1, 1000) i;
     CHECKPOINT;
     SET wal_compression = pglz;
     UPDATE w SET b = 'y' WHERE a < 100;
   ```
     Actual: the UPDATE succeeds, but all segment mirrors stop WAL replay and shut down, and FTS marks them down:
     FATAL:  invalid compressed image at 0/C7CFF070, block 0 (error computing decompression size)
     LOG:    startup process (PID ...) exited with exit code 1
     LOG:    shutting down due to startup process failure
     gprecoverseg (incremental) reports success, but the mirrors replay the same WAL record and crash again. Only gprecoverseg -F (full recovery) brings them back.
   
     Expected: the mirrors replay the WAL normally, as they do with SET wal_compression = zstd. With zstd, the same steps produce no errors.
   
     Cause: in RestoreBlockImage() (src/backend/access/transam/xlogreader.c, around line 2096), every compressed page image is passed to zstd_decompress_backupblock() before the code checks bimg_info for the method
     actually used. That zstd-only call dates from commit 60915ec6b2e ("Use zstd for wal_compression", 2020), when Greenplum always compressed WAL with zstd. Since the merges from PostgreSQL,
     XLogCompressBackupBlock() writes images with the method chosen by wal_compression (pglz, lz4 or zstd), so pglz and lz4 images reach the zstd decoder and fail. A likely fix is to remove the unconditional zstd
     call and rely on the per-method code that follows it.
   
     Impact:
     - A session-level SET wal_compression plus an ordinary write is enough. It leaves the cluster without mirror protection until an administrator runs a full recovery.
     - Not verified: lz4 (expected to fail the same way, since the same code runs first), and whether a primary's own crash recovery also fails after replaying such a record.
   
   
   ### What you think should happen instead
   
   _No response_
   
   ### How to reproduce
   
     Reproduce (on a cluster with mirrors):
   ```sql
     CREATE TABLE w (a int, b text) DISTRIBUTED BY (a);
     INSERT INTO w SELECT i, repeat('x', 100) FROM generate_series(1, 1000) i;
     CHECKPOINT;
     SET wal_compression = pglz;
     UPDATE w SET b = 'y' WHERE a < 100;
   ```
   
   ### Operating System
   
   any
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes, I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org