You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@answer.apache.org by "lhk0504 (via GitHub)" <gi...@apache.org> on 2026/09/22 08:29:55 UTC
[I] CleanOrphanUploadFiles may skip records when using offset pagination [answer]
lhk0504 opened a new issue, #1631:
URL: https://github.com/apache/answer/issues/1631
## Describe the bug
`CleanOrphanUploadFiles` may skip some available file records during a cleanup run because it uses offset-based pagination while modifying the same result set being paginated.
The cleanup job queries file records with `status = Available` in batches of 1000. While processing a batch, orphan files are cleaned by `DeleteAndMoveFileRecord`, which changes their status from `Available` to `Deleted`.
Because those records no longer match the `status = Available` condition, the result set becomes smaller. However, the next iteration increments the page number and continues using an offset calculated from the new result set.
For example, if there are 1001 eligible orphan records:
* Page 1 queries the first 1000 records with `OFFSET 0`.
* Those 1000 records are changed from `Available` to `Deleted`.
* Only 1 `Available` record remains.
* Page 2 queries with `OFFSET 1000`.
* The remaining record is skipped, so the query returns no records and the cleanup loop exits.
As a result, not all eligible orphan file records are guaranteed to be checked in a single cleanup run. The skipped records remain `Available` and may only be processed by a later cleanup run.
## To Reproduce
Steps to reproduce the behavior:
1. Create 1001 file records with:
* `status = Available`
* `CreatedAt` older than 2 days
* no revision or other object referencing the uploaded file
* corresponding uploaded files present on disk
2. Run `CleanOrphanUploadFiles`.
3. Query the remaining file records with `status = Available`.
4. Observe that at least one eligible orphan record remains unprocessed.
The issue comes from combining offset pagination with updates to the filtered result set:
```text
First query:
OFFSET 0 LIMIT 1000
→ 1000 records are processed
→ their status changes from Available to Deleted
Remaining Available records:
1
Second query:
OFFSET 1000 LIMIT 1000
→ returns no records
→ cleanup exits
```
## Expected behavior
`CleanOrphanUploadFiles` should scan all file records that are eligible for the current cleanup run, even when previously scanned records are changed from `Available` to `Deleted`.
No eligible record should be skipped because earlier records were removed from the `Available` result set during the same cleanup execution.
## Screenshots
Not applicable.
## Platform
* Device: N/A
* OS: Not OS-specific
* Browser and version: N/A
* Version: current `main` (`3b9f1370612e690a0b7f230f05e688930db4c6d3`)
* Deployment method: Source
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For additional commands, e-mail: issues-help@answer.apache.org
Re: [I] CleanOrphanUploadFiles may skip records when using offset pagination [answer]
Posted by "lhk0504 (via GitHub)" <gi...@apache.org>.
lhk0504 closed issue #1631: CleanOrphanUploadFiles may skip records when using offset pagination
URL: https://github.com/apache/answer/issues/1631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For additional commands, e-mail: issues-help@answer.apache.org
Re: [I] CleanOrphanUploadFiles may skip records when using offset pagination [answer]
Posted by "Asthenia0412 (via GitHub)" <gi...@apache.org>.
Asthenia0412 commented on issue #1631:
URL: https://github.com/apache/answer/issues/1631#issuecomment-6156245123
Hi @yancy here (Asthenia0412). I've opened PR #1642 to address this bug.
## Root cause
The cron function uses offset-based pagination () while simultaneously deleting orphan records that drop out of the filter. After a batch deletes N records, the remaining result set shrinks and records that were at positions and above slide into earlier slots. Advancing the page counter then skips past those shifted rows.
## Fix
Track counter per iteration. When deletions occur, reset to so the next query rescans the front of the current Available set. When no deletions occur the Available set is stable and incrementing is safe.
## Verification
- `go build ./...` ✅ full project builds cleanly
- Fix scenario from the issue description (1001 orphan records): now correctly processes all records instead of leaving 1 orphan unprocessed
PR link: https://github.com/apache/answer/pull/1642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For additional commands, e-mail: issues-help@answer.apache.org
Re: [I] CleanOrphanUploadFiles may skip records when using offset pagination [answer]
Posted by "LinkinStars (via GitHub)" <gi...@apache.org>.
LinkinStars commented on issue #1631:
URL: https://github.com/apache/answer/issues/1631#issuecomment-6162520366
Thanks for the fix! Resetting to the first page after records are changed does address the original offset-pagination skip.
One edge case may still be worth covering: if the first 1,000 available records are all retained (for example, they are still referenced, within the 48-hour window, or active branding/avatar files), `changed` remains `false` and the loop exits. Any orphan records on later pages would then not be examined during that run.
Could we add a regression test for that case and continue to the next page when a stable page makes no changes? A hybrid approach could reset to page 1 after a deletion/update, but advance to the next page when no record changes. This preserves the offset fix while ensuring later pages are still scanned.
Thanks again for working on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For additional commands, e-mail: issues-help@answer.apache.org
Re: [I] CleanOrphanUploadFiles may skip records when using offset pagination [answer]
Posted by "lhk0504 (via GitHub)" <gi...@apache.org>.
lhk0504 commented on issue #1631:
URL: https://github.com/apache/answer/issues/1631#issuecomment-5773817773
Hi @LinkinStars , could you please take a look at this issue when you have time? I found that `CleanOrphanUploadFiles` may skip records because it uses offset-based pagination while modifying the same filtered result set.
I'd also be interested in working on a fix if the issue is confirmed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@answer.apache.org
For additional commands, e-mail: issues-help@answer.apache.org