You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "garvit-arora (via GitHub)" <gi...@apache.org> on 2026/10/13 02:35:13 UTC
[PR] feat(storage): reclaim free runtime SQLite pages in Host maintenance [maka]
garvit-arora opened a new pull request, #6083:
URL: https://github.com/apache/maka/pull/6083
## Summary
Add a bounded Host maintenance lane that returns free pages from new `runtime.sqlite` databases initialized with `auto_vacuum = INCREMENTAL` by #6044. Each tick reads `auto_vacuum` and `freelist_count` in a read transaction, skips older `NONE` files without a write lock, runs at most 64 pages of `incremental_vacuum` through the process-local operational database owner, and follows a productive commit with a PASSIVE checkpoint. The lane uses the existing 100 ms active and 60 s idle cadence and drains with Host maintenance.
Existing `NONE` databases remain report-only. This PR does not convert them, add another SQLite owner connection, or change protocol, UI, or compatibility epoch.
Refs #6000
## Verification
- Core, storage, MCP, runtime, and Runtime Host builds passed on the rebased head.
- Biome lint passed for the five changed files; Biome format passed after the one formatting correction.
- No test suite was run or added in this draft. Focused reclamation and maintenance tests remain before this PR is ready for review.
## AI use
- [ ] No generative tool made a substantive contribution
- [x] Generative tooling made a substantive contribution
Tool(s) and scope: Codex helped implement and review the storage and maintenance lane.
## Checklist
- [ ] Tests cover the change and fail without it
- [ ] Lint, format, typecheck and the affected suites pass locally
Does this PR entail a change in behavior?
- [x] Yes — new incremental runtime databases can reclaim free pages in Host maintenance.
- [ ] No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(storage): reclaim free runtime SQLite pages in Host maintenance [maka]
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on PR #6083:
URL: https://github.com/apache/maka/pull/6083#issuecomment-6248389159
@garvit-arora I opened #6149 for the context-offload half of the checkpoint finding, so it stays tracked even if this PR keeps to runtime.sqlite. A shared helper that fixes both lanes is still the preferred route; if you take that route, add `Fixes #6149` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(storage): reclaim free runtime SQLite pages in Host maintenance [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #6083:
URL: https://github.com/apache/maka/pull/6083#discussion_r4270218011
##########
packages/storage/src/runtime-page-reclamation.ts:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import {
+ acquireOperationalStateDatabase,
+ type OperationalStateDatabaseLease,
+} from './operational-state-store.js';
+import { runWithStorageRootLease, type StorageRootLease } from './root-authority.js';
+
+export interface RuntimePageReclamation {
+ reclaimFreePages(input: { readonly maxPages: number }): Promise<{
+ readonly reclaimedPages: number;
+ readonly hasMore: boolean;
+ }>;
+}
+
+/** Uses the process-local operational owner; older NONE databases remain report-only. */
+export function openRuntimePageReclamation(
+ rootLease: StorageRootLease<'interactive', 'write'>,
+): RuntimePageReclamation {
+ return Object.freeze({
+ reclaimFreePages: ({ maxPages }: { readonly maxPages: number }) => {
+ if (!Number.isSafeInteger(maxPages) || maxPages <= 0) {
+ throw new Error('Runtime page reclamation limit must be a positive safe integer');
+ }
+ return runWithStorageRootLease(rootLease, 'interactive', 'write', async (root) => {
+ const databaseLease = acquireOperationalStateDatabase(root, {
+ schemaMigration: 'require_current',
+ });
+ try {
+ return reclaimFreePages(databaseLease, maxPages);
+ } finally {
+ databaseLease.close();
+ }
+ });
+ },
+ });
+}
+
+function reclaimFreePages(
+ lease: OperationalStateDatabaseLease,
+ maxPages: number,
+): { readonly reclaimedPages: number; readonly hasMore: boolean } {
+ // An idle tick and an older NONE database must never acquire a write lock.
+ const eligible = lease.transaction('read', () => {
+ const mode = readPragma(lease, 'auto_vacuum');
+ if (mode !== 2) return false;
+ return readPragma(lease, 'freelist_count') > 0;
+ });
+ if (!eligible) return { reclaimedPages: 0, hasMore: false };
Review Comment:
[P2] Retry an unfinished checkpoint after the freelist drains
PASSIVE may leave frames uncheckpointed while another connection holds an old read snapshot. If that overlaps the final vacuum batch, the freelist reaches zero and this return prevents every subsequent idle call from finishing physical truncation. In a real SQLite reproduction, 536 pages were reclaimed logically, yet runtime.sqlite remained 3,334,144 bytes after the reader committed and the next idle API call completed; one extra PASSIVE checkpoint immediately reduced it to 1,138,688 bytes. The idle-physical-shrink assertion fails here and passes with a diagnostic checkpoint retry. Preserve a non-blocking completion path after the freelist drains, while keeping old NONE files untouched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org