You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudberry.apache.org by "my-ship-it (via GitHub)" <gi...@apache.org> on 2026/04/24 02:20:10 UTC

[PR] Fix: gprestore --resize-cluster fails with --jobs > 1 [cloudberry-backup]

my-ship-it opened a new pull request, #93:
URL: https://github.com/apache/cloudberry-backup/pull/93

   ## Summary
   
   Fixes #92.
   
   `gprestore --resize-cluster` (larger-to-smaller) only works with `--jobs 1` today. Any `--jobs > 1` run fails with:
   
   ```
   ERROR: command error message: cat: .../gpbackup_<c>_<ts>_pipe_..._<oid>_0: No such file or directory
   ```
   
   This is the same class of bug upstream gpbackup fixed in 1.30.6 (and documented in [Broadcom KB 378496](https://knowledge.broadcom.com/external/article/378496/gprestore-with-resizecluster-will-fail-w.html)); the pre-fix logic is still present in this fork.
   
   ## Root cause
   
   In `restore/data.go:restoreDataFromTimestamp`, the oid list for resize restore is expanded into `(oid, batch)` pairs in oid-major order:
   
   ```
   [T1B0, T1B1, T2B0, T2B1, ..., TnB0, TnB1]
   ```
   
   `CreateInitialSegmentPipes` then preloads only `min(NumConns, len(oidList))` pipes, and `gpbackup_helper`'s `preloadCreatedPipesForRestore` mirrors that count via `--copy-queue-size`. With `NumConns == --jobs` and `batches ≥ 2`, the preload covers only the first `NumConns/batches` tables.
   
   But the coordinator's task channel dispatches **one task per table**, and `restoreSingleTableData` iterates the batches sequentially inside each worker. So `NumConns` concurrent workers each immediately open a COPY against oid indices `0, batches, 2*batches, …, (NumConns-1)*batches`. Workers beyond `NumConns/batches` request pipes the helper has not created yet — hence the "No such file or directory" error.
   
   With `--jobs 1` the coordinator serializes completely and the helper keeps up, which matches the reporter's observation.
   
   ## Fix
   
   Preload `NumConns * batches` pipes (clamped to `len(oidList)`), so every concurrent worker's first batch is covered. The helper then rolls its queue forward one pipe per completed batch, and because a worker must finish **all** batches of its current table before pulling the next task from the channel, helper progress stays ahead of worker demand.
   
   - `batches == 1` (non-resize or equal-size resize): `NumConns * 1 == NumConns`, behavior unchanged.
   - `batches > 1`: each of the `NumConns` concurrent workers has its first batch preloaded; subsequent tables are served from the rolling queue.
   
   No change to the helper — `preloadCreatedPipesForRestore` and the `if i < len(oidWithBatchList)-*copyQueue` guard work with any queue size.
   
   ## Test plan
   
   - [x] `go build ./...` passes
   - [x] `go vet ./restore/...` clean
   - [x] `gofmt -l restore/data.go` clean
   - [ ] Manual repro on a 6→4 segment resize restore with `--jobs 8` (pre-patch: errors on every table; post-patch: restore completes)
   - [ ] CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix: gprestore --resize-cluster fails with --jobs > 1 [cloudberry-backup]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe merged PR #93:
URL: https://github.com/apache/cloudberry-backup/pull/93


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix: gprestore --resize-cluster fails with --jobs > 1 [cloudberry-backup]

Posted by "tuhaihe (via GitHub)" <gi...@apache.org>.
tuhaihe commented on PR #93:
URL: https://github.com/apache/cloudberry-backup/pull/93#issuecomment-4311863890

   Hi @woblerr could you help review this PR? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org


Re: [PR] Fix: gprestore --resize-cluster fails with --jobs > 1 [cloudberry-backup]

Posted by "woblerr (via GitHub)" <gi...@apache.org>.
woblerr commented on PR #93:
URL: https://github.com/apache/cloudberry-backup/pull/93#issuecomment-4314478201

   Hi, LGTM.
   As a non-blocking improvement, add an extra e2e test for case  `--resize-cluster` with `--jobs > 1`.
   
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cloudberry.apache.org
For additional commands, e-mail: commits-help@cloudberry.apache.org