You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "liugddx (via GitHub)" <gi...@apache.org> on 2026/10/18 09:45:12 UTC
[I] bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint [maka]
liugddx opened a new issue, #6149:
URL: https://github.com/apache/maka/issues/6149
### What happens
`reclaimFreePages` in `packages/storage/src/sqlite-context-offload-store.ts` (added in #5855) runs `PRAGMA wal_checkpoint(PASSIVE)` only in the call where `incremental_vacuum` reclaimed pages:
```ts
if (this.#readTransaction(() => this.#freelistPages()) === 0) {
return { reclaimedPages: 0, hasMore: false };
}
// ...
if (result.reclaimedPages > 0) this.#database.exec('PRAGMA wal_checkpoint(PASSIVE)');
```
PASSIVE does not wait for readers. If another connection holds an older read snapshot while the last batch commits, the checkpoint stops early and the truncated pages stay in the WAL. The freelist is already empty at that point, so every later maintenance tick returns at the first check and never retries the checkpoint. The pages are reclaimed logically, but `context-offload.sqlite` does not shrink on disk until some unrelated write happens to trigger an autocheckpoint. On an idle install, that may never happen.
@Astro-Han reproduced the same pattern with real SQLite in the review of #6083, which copies this logic for `runtime.sqlite`: after 536 pages were drained with a reader pinned, the file stayed at 3,334,144 bytes; one extra PASSIVE checkpoint brought it to 1,138,688 bytes.
This is not data corruption: reads, the freelist accounting and `quick_check` are all correct. The problem is only that the disk space is not returned.
### Expected
Once the freelist drains, a checkpoint that could not finish is retried later without blocking. The maintenance lane still never takes the write lock on an idle tick, never waits on readers, and never runs a full `VACUUM`.
### Suggested fix
Read the row that `wal_checkpoint(PASSIVE)` returns (`busy`, `log`, `checkpointed`). If `busy = 1` or `checkpointed < log`, keep an in-memory "checkpoint pending" flag. On the next tick, if the freelist is empty but the flag is set, run PASSIVE again outside any write transaction, and clear the flag once it completes. Return `hasMore: false` so the lane stays on its idle cadence instead of spinning while a reader holds its snapshot.
Add a regression test that holds a read transaction across the final batch, releases it, and asserts that the file shrinks on the next idle call.
Ideally `runtime.sqlite` (#6083) and context-offload share one small helper, so the two lanes do not drift apart. See my review on #6083.
Refs #5855, #6083, #6000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint [maka]
Posted by "Dante-dan (via GitHub)" <gi...@apache.org>.
Dante-dan commented on issue #6149:
URL: https://github.com/apache/maka/issues/6149#issuecomment-6248860550
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint [maka]
Posted by "Dante-dan (via GitHub)" <gi...@apache.org>.
Dante-dan commented on issue #6149:
URL: https://github.com/apache/maka/issues/6149#issuecomment-6248859726
Automated contribution plan, prepared with OpenAI Codex:
The empty-freelist return currently skips the only PASSIVE checkpoint path, so a checkpoint interrupted by a pinned reader loses its retry opportunity. I plan to keep the checkpoint completion state in memory, read the `busy`/`log`/`checkpointed` result, and retry a pending PASSIVE checkpoint on later idle calls after the freelist drains. The retry stays outside write transactions and returns `hasMore: false` so readers cannot cause a busy maintenance loop.
I will put the checkpoint-state logic in a small internal storage helper that the runtime lane can reuse, while limiting this change to context-offload: #6083 currently remains runtime-only. I will add the requested real-SQLite regression holding a read transaction across the final reclamation batch, then releasing it and checking that the next idle call shrinks the main file, alongside the existing idle-lock and bounded-batch coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han closed issue #6149: bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint
URL: https://github.com/apache/maka/issues/6149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [I] bug(storage): context-offload page reclamation never retries an unfinished WAL checkpoint [maka]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #6149:
URL: https://github.com/apache/maka/issues/6149#issuecomment-6250879755
Thanks @Dante-dan, assigned. The plan matches what I had in mind: keep the checkpoint state in memory, retry PASSIVE outside write transactions after the freelist drains, return `hasMore: false`, and add the pinned-reader regression. Keeping the helper reusable but scoped to context-offload is fine. #6083 can adopt it for `runtime.sqlite` once both land. Please say `Fixes #6149` in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org