You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by "MicroGery (via GitHub)" <gi...@apache.org> on 2026/09/23 02:16:15 UTC
[PR] feat(runtime-host): publish verified local session checkpoints [maka]
MicroGery opened a new pull request, #5605:
URL: https://github.com/apache/maka/pull/5605
## Summary
Refs #2370 — PR2 of the September 9 local Runtime follow-up, after #5090.
- Add opt-in Host checkpoint publication through the selected persistence provider: fence the complete private copy, verify Bundle and Manifest, then create/CAS the Repository Head. Live SQLite commits and ordinary restart remain unchanged.
- Persist stable bindings, frozen request inputs and receipts so crashes, lost replies and retries do not recapture or advance a revision twice. Report terminal publication separately from recoverable temporary-file cleanup.
- Add real Host crash/codec integration, provider/maintenance fencing and independent-process Repository race tests. This does not implement PR3 restore/activation.
## Verification
- Clean `npm run build:test`, then rebuild affected packages after the review fixes.
- All workspace tests, serial workspace scheduling: JS/TS **12,939 passed / 38 skipped / 0 failed**; Python 3.12 **75 passed / 12 skipped**. Host: 2,034 passed; Storage: 1,418 passed.
- Focused checkpoint/crash/Repository tests: **38/38**; the two new cleanup regression tests were also verified failing before the fix.
- Full typecheck, lint, format check, Desktop/UI knip, all 21 PR files' ASF headers, Windows skip inventory (100 declarations), and diff whitespace checks passed.
- Executed on macOS arm64 / Node 24.18.0. Native Windows/Linux execution and CI results are not claimed here.
## Rollout and review focus
Disabled by default; no protocol, UI or automatic checkpoint hook. Enabling requires a trusted workspace authority that fences **all** writers, including external processes. Arbitrary developer directories are not assumed safe. Windows additionally requires a private-staging ACL authority; the Memory reference provider explicitly reports unsupported.
Adversarial review reproduced and fixed two cleanup failure paths: a committed checkpoint reported as failed when deleting its temporary archive failed, and conflicted archives retaining no retry cleanup path. Both regression tests failed before the fix and passed afterward. Prepared inputs and immutable objects are never part of terminal archive cleanup.
## Remaining validation
- [ ] Record repeated same-machine fence/phase timings, peak memory and disk usage for representative workloads, as requested in the follow-up plan. Functional suite durations are not a performance baseline.
## AI use
- [ ] No generative tool made a substantive contribution
- [x] Generative tooling made a substantive contribution
Tool(s) and scope: OpenAI Codex implemented the change, performed this self-review, wrote regression tests and prepared this submission at the contributor's request. Both commits include `Generated-by: Codex`. Independent human review is still required.
## Checklist
- [x] Tests cover the change and fail without it
- [x] Lint, format, typecheck and the affected suites pass locally
Does this PR entail a change in behavior?
- [x] Yes — described under Summary above
- [ ] No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "Astro-Han (via GitHub)" <gi...@apache.org>.
Astro-Han commented on code in PR #5605:
URL: https://github.com/apache/maka/pull/5605#discussion_r4303185997
##########
scripts/asf-license-headers.mjs:
##########
@@ -110,6 +110,7 @@ const commentStyles = {
*/
const coveredExtensions = new Map([
['.astro', 'astro'],
+ ['.c', 'block'],
Review Comment:
[P2] This covers `.c` in the header audit, but `scripts/asf-source-release.mjs` `textSourceExtensions` has no `.c` entry. The CI "ASF source release" step therefore fails with `Cannot safely classify non-text release input .../macos-checkpoint-capability-probe.c`. Adding `'.c'` to that set (with a test) fixes it locally. Alternatively, drop the diagnostic probe from this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "MicroGery (via GitHub)" <gi...@apache.org>.
MicroGery merged PR #5605:
URL: https://github.com/apache/maka/pull/5605
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "likun666661 (via GitHub)" <gi...@apache.org>.
likun666661 commented on PR #5605:
URL: https://github.com/apache/maka/pull/5605#issuecomment-5788429280
Review note: This is a well-bounded PR2 for opt-in, verified local checkpoint publication. The Bundle -> Manifest -> Head ordering, expected-revision CAS, and durable prepared-request record give a coherent answer to stale writers, crashes, and lost replies. Keeping restore/activation, automatic publication, and arbitrary externally writable workspaces out of scope is the right boundary.
Before treating this as deployable, I would make the trusted `HostCheckpointWorkspaceAuthority` the primary review gate: identify the concrete deployment implementation and demonstrate that its fence covers all writers to the workspace, including independent/external processes, for the entire private-copy interval. The interface contract and an in-process test authority alone cannot establish that property. Please also retain explicit unsupported behavior when this guarantee is unavailable, and validate root-wide provider-fence latency under representative contention on the intended platforms; the current macOS measurements are a useful baseline, not a cross-platform or automatic-publication SLO.
I view this as convergence on the PR2 publication problem, not closure of the end-to-end Session recovery problem. Exact checkout, isolated hydrate, and safe activation still need PR3 validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "hqhq1025 (via GitHub)" <gi...@apache.org>.
hqhq1025 commented on code in PR #5605:
URL: https://github.com/apache/maka/pull/5605#discussion_r4106064959
##########
packages/runtime-host/src/server/session-checkpoint-publication.ts:
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { mkdir, realpath, rm } from 'node:fs/promises';
+import { join } from 'node:path';
+import type { SessionManager } from '@maka/runtime/session-manager';
+import { SessionConfigurationTransitionError } from '@maka/runtime/session-manager';
+import type { InteractiveExecutionStoresWriter } from '@maka/storage/execution-stores';
+import { openFileSessionRepository } from '@maka/storage/file-session-repository';
+import { openSessionCheckpointPublicationStore } from '@maka/storage/session-checkpoint-publication-store';
+import { acquireProcessLifetimeOwner } from '@maka/storage/process-lifetime-owner';
+import { runWithStorageRootLease } from '@maka/storage/root-authority';
+import {
+ assertSessionBundleLimits,
+ type SessionBundleLimits,
+} from '@maka/storage/session-bundle-contract';
+import { createProductionSessionSnapshotService } from '@maka/storage/production-session-snapshot';
+import type {
+ SessionSnapshotCancellation,
+ SessionSnapshotPrivateStagingRootAuthority,
+ SessionSnapshotStatePreparer,
+ SessionSnapshotWorkspaceConfirmationAuthority,
+} from '@maka/storage/quiescent-session-snapshot';
+import type { InteractiveShellRunWriter } from '@maka/storage/shell-run-authority';
+import type { RuntimeHostCompositionContext } from './host-kernel.js';
+import type { SessionAdmissionGate } from './session-admission-gate.js';
+import {
+ HostCheckpointError,
+ HostSessionCheckpointCoordinator,
+ type HostSessionCheckpointPublication,
+} from './session-checkpoint-coordinator.js';
+
+/**
+ * Trusted deployment authority, never a client/model supplied "idle" flag.
+ * It must fence ALL writers (including external processes/shared workspaces)
+ * before invoking operation, and hold that fence until the copy settles.
+ * Ordinary externally writable directories without this authority are unsupported.
+ */
+export interface HostCheckpointWorkspaceAuthority {
+ runExclusive<T>(
+ input: {
+ readonly sessionId: string;
+ readonly workspaceRoot: string;
+ readonly cancellation: SessionSnapshotCancellation;
+ },
+ operation: () => Promise<T>,
+ ): Promise<T>;
+}
+export interface HostCheckpointPublicationOptions {
+ readonly limits: SessionBundleLimits;
+ readonly workspace: HostCheckpointWorkspaceAuthority;
+ readonly privateStagingRootAuthority?: SessionSnapshotPrivateStagingRootAuthority;
+ readonly confirmationAuthority?: SessionSnapshotWorkspaceConfirmationAuthority;
+}
+
+/** Trusted service capability; no protocol endpoint or model tool is added by PR2. */
+export async function createHostSessionCheckpointPublication(input: {
+ readonly context: RuntimeHostCompositionContext;
+ readonly stores: InteractiveExecutionStoresWriter;
+ readonly admission: SessionAdmissionGate;
+ readonly manager: SessionManager;
+ readonly shellRuns: InteractiveShellRunWriter;
+ readonly isSessionIdle: (sessionId: string) => boolean;
+ readonly options?: HostCheckpointPublicationOptions;
+}): Promise<HostSessionCheckpointPublication> {
+ const { context, stores, options } = input;
+ if (!options) return unavailable('Checkpoint publication is not configured');
+ if (!stores.snapshot)
+ return unavailable('Selected execution persistence does not support checkpoints');
+ if (!options.workspace || typeof options.workspace.runExclusive !== 'function') {
+ return unavailable('A trusted workspace snapshot authority is required');
+ }
+ if (process.platform === 'win32' && !options.privateStagingRootAuthority) {
+ return unavailable('Checkpoint staging requires a Windows ACL verification authority');
+ }
+ assertSessionBundleLimits(options.limits);
+ // Freeze caller policy at composition, not midway through a publication.
+ const limits = Object.freeze({ ...options.limits });
+ const root = join(context.owner.controlDirectory, 'session-checkpoints-v1');
+ const stagingParent = join(root, 'staging');
+ const cleanupStateRoot = join(root, 'cleanup');
+ const bundlesRoot = join(root, 'bundles');
+ let owner: Awaited<ReturnType<typeof acquireProcessLifetimeOwner>> | undefined;
+ try {
+ await Promise.all(
+ [stagingParent, cleanupStateRoot, bundlesRoot].map((path) =>
+ mkdir(path, { recursive: true, mode: 0o700 }),
+ ),
+ );
+ const cleanupOwner = await acquireProcessLifetimeOwner(cleanupStateRoot);
Review Comment:
[P2] Recover private staging after acquiring the cleanup owner and before opening checkpoint publication. `createProductionSessionSnapshotService` calls `stagingCleanup.recover()`, but this Host path constructs that service only inside a later `capture`. After a crash leaves a private staging file, restarting the Host without another capture (including merely replaying a committed receipt) never scans the persisted cleanup lease, so that file can remain indefinitely. Please add a crash → restart → no further capture cleanup regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "MicroGery (via GitHub)" <gi...@apache.org>.
MicroGery commented on PR #5605:
URL: https://github.com/apache/maka/pull/5605#issuecomment-6334344879
@Astro-Han The requested split is now pushed.
- **#5605:** removal commit `d95f8c0f2` removes the three macOS probe files, their usage paragraphs, and the probe-only C header-policy additions. The publication implementation, opt-in/unsupported behavior and journal semantics are unchanged.
- **Separate probe PR:** #6233, based directly on main, contains the diagnostic and standalone documentation. It also adds C source-release classification and a regression test, so the packaging omission does not follow the probe into its new PR.
For the #5605 removal tree, actual source-archive creation/verification passed locally; header/source-release tests passed **65/65**, and checkpoint publication/coordinator tests passed **31/31**. The new required CI is running; this is not a claim that it has completed.
For #6233, diagnostic/header/source-release tests passed **72/72**, the new C archive test was verified failing before the fix, the native C syntax check passed, and a real source archive from its committed head passed verification. No new disk-image operations or privileges were used during the split validation.
The concrete workspace authority, workspace consistency contract and journal compaction remain follow-up work as discussed. The extracted probe is diagnostic-only and does not supply a production authority.
*Prepared with Codex assistance.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "MicroGery (via GitHub)" <gi...@apache.org>.
MicroGery commented on PR #5605:
URL: https://github.com/apache/maka/pull/5605#issuecomment-6230741834
@likun666661 Following up on your [workspace-authority review note](https://github.com/apache/maka/pull/5605#issuecomment-5788429280): could we discuss which concrete deployment model to target before expanding `HostCheckpointWorkspaceAuthority`?
The local-use goal is to keep working in the **existing project directory**, without requiring users to move it into a Maka-managed disk-image volume. The requirement is a stable workspace view aligned with the captured Session state—not necessarily freezing the live directory if a real read-only snapshot provides that view.
There are four approaches, with different prerequisites and guarantees:
| Approach | How it works | Prerequisites and limitations |
| --- | --- | --- |
| **1. Maka creates a native APFS snapshot** | A native helper creates a fresh filesystem snapshot; capture reads the project's subtree from a read-only view. | The syscall path requires a snapshot entitlement and the necessary local privileges. A local, unpublished probe on two disposable APFS volumes confirmed filesystem snapshot support, but the unentitled process received `EPERM`. We have not established entitlement eligibility or a distributable signing setup. Administrator privileges alone do not bypass the entitlement check. |
| **2. Use macOS-provided snapshot tools/services** | Investigate `tmutil` local snapshots and `mount_apfs -s` as the source of the stable read view, potentially avoiding a snapshot entitlement on Maka's own executable. | **Candidate only; no end-to-end validation yet.** It may still need local administrator/privacy permissions and Time Machine configuration. The installed `tmutil` manual says creation affects all APFS volumes included in the backup, not a selected project directory. Snapshot identity, automatic reclamation, mount lifetime, latency and safe cleanup must be validated; we must not delete unrelated backups. |
| **3. Cooperatively quiesce every writer, then copy** | Stop new writes, drain in-flight writes, hold exclusion through the private copy, then resume. The original project path stays unchanged. | No APFS snapshot entitlement, but reliable only where **every actual writer participates or is otherwise reliably excluded**. Pausing Maka alone, adding an advisory lock, or observing an idle directory does not stop an independent editor, shell or build process. This is a controlled-deployment option, not a general guarantee for arbitrary developer directories. |
| **4. Bind Session state to an explicit immutable workspace version** | Capture a declared Git tree/content-addressed manifest, retain its objects, and record the exact version binding with the Session checkpoint. | No APFS snapshot entitlement. However, this changes what we promise: an exact declared version is not automatically the current complete working directory, including dirty/untracked files. Capturing those files under arbitrary concurrent writes remains unsolved. This would need an explicit scope/contract decision, not a silent fallback. |
For approaches 1 and 2, “volume-level” means the underlying snapshot covers the filesystem volume even though the Bundle includes only the authorized project subtree. The original project is neither moved nor rolled back, but snapshot resource usage and lifecycle can extend beyond that project.
Whichever model we choose, the existing Host/Runtime/provider fences still need to align Session state with the chosen workspace view. Snapshot-backed capture also needs a real read-view handoff: creating a snapshot while the copier continues reading the live path would not satisfy the requirement. Bundle → Manifest → Head/CAS ordering, fixed-input retries and explicit unsupported behavior should remain intact. A filesystem point-in-time view also does not certify transactions performed by unrelated applications.
My tentative preference is to **validate approach 2 before committing to an Apple-entitlement dependency**, without making Time Machine a default dependency or assuming that this candidate works. If its volume-wide effects or operational requirements are unacceptable, we should explicitly choose a controlled-writer deployment (3) or revisit the workspace-version contract (4).
What do you think about:
- Whether full live-workspace point-in-time capture should remain the required local checkpoint contract, versus an explicitly bound immutable version.
- Whether a concrete supported authority should land in this PR or a separate platform adapter, with this PR remaining disabled/unsupported where no such authority is available.
The probe is local feasibility work, not a production authority or a change already included in the current PR head. These alternatives do not implement PR3 restore/activation.
References: [Apple snapshot entitlement check](https://github.com/apple-oss-distributions/xnu/blob/f6217f891ac0bb64f3d375211650a4c1ff8ca1ea/bsd/vfs/vfs_syscalls.c#L15215), [Time Machine snapshot lifecycle](https://support.apple.com/en-ca/102154), and [Git tree/index semantics](https://git-scm.com/docs/git-write-tree).
*Prepared with Codex assistance; deployment options and scope are proposed for review, not established community decisions.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org
Re: [PR] feat(runtime-host): publish verified local session checkpoints [maka]
Posted by "MicroGery (via GitHub)" <gi...@apache.org>.
MicroGery commented on PR #5605:
URL: https://github.com/apache/maka/pull/5605#issuecomment-6264491795
@Astro-Han Thanks for the review. Could you also weigh in on the workspace-authority options we raised with Kun in [this earlier comment](https://github.com/apache/maka/pull/5605#issuecomment-6230741834)?
The goal is to keep using the existing project directory while obtaining a stable workspace view aligned with the captured Session state. The four options under discussion are:
1. **Native APFS snapshot helper.** Capture from a read-only filesystem snapshot. This route has entitlement/signing and local-privilege prerequisites; our unentitled disposable-volume probe returned `EPERM`. We have not established a deployable helper.
2. **macOS-provided snapshot tools/services.** Investigate `tmutil` and `mount_apfs -s` to obtain the stable read view without putting the snapshot entitlement on Maka's own executable. This is still an unvalidated candidate, not a working adapter. Permissions, volume-wide effects, snapshot identity/reclamation, mounting and safe cleanup all need validation.
3. **Cooperative quiescence of all writers.** Drain and exclude writes for the entire private copy. This only works in a controlled deployment where every writer participates or is reliably excluded; pausing Maka alone cannot stop an external editor, shell or build process.
4. **An explicitly bound immutable workspace version.** Bind the Session checkpoint to a retained Git tree/content-addressed manifest. This changes the contract: it captures an exact declared version, not automatically the full live working directory including dirty/untracked files. It must not become a silent fallback.
For snapshot-backed options, the copier must actually read the stable snapshot view, aligned with the Session-state fence, rather than continue reading the live path. The existing Bundle → Manifest → Head/CAS ordering and fixed-input retry guarantees should remain unchanged.
Two scope questions:
- Should local checkpoints require a complete live-workspace point-in-time view, or would an explicitly declared immutable version be a useful separately specified contract?
- Would you support keeping the concrete platform authority as an explicit follow-up, with #5605 remaining opt-in and unavailable without a trustworthy authority? Or should one deployable authority be included before this PR lands? Your suggestion to split out the diagnostic probe seems relevant to that boundary.
One update to the earlier comment: the probe is now included in `ee5bdd6b1`, but remains diagnostic-only, not a production authority. The source-release P2 you identified is a separate concrete defect; this design discussion does not resolve or excuse it.
*Prepared with Codex assistance; these are proposals for discussion, not agreed implementation or merge decisions.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org