You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by as...@apache.org on 2026/10/11 15:16:36 UTC
(maka) branch main updated: ci(perf): add a Host startup profile measurement script (#5791)
This is an automated email from the ASF dual-hosted git repository.
Astro-Han pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/maka.git
The following commit(s) were added to refs/heads/main by this push:
new 6b9457f5bb ci(perf): add a Host startup profile measurement script (#5791)
6b9457f5bb is described below
commit 6b9457f5bb3c1af96e65ec9a12cc6bae63f49aaa
Author: 小七 <gg...@gmail.com>
AuthorDate: Sun Oct 11 23:16:30 2026 +0800
ci(perf): add a Host startup profile measurement script (#5791)
#4677's "re-profile Host startup" acceptance item lacks a current
baseline because the startup anchors (composition open, five-phase
domain recovery, first catalog query, orderly close) had no repeatable
local measurement. This adds a self-contained runtime-host script that
seeds a usage-shaped fixture (Sessions x Turns x artifacts), reopens
the composition for the configured number of rounds, and reports
per-anchor medians per restart.
Medians measured on this head's ancestor fffc19fb5 (Windows 11,
Node 26.10.0, one machine for all rounds):
| fixture | open | recover | firstQuery | close | restart total |
| --------------------------------------------- | ---: | ------: | ---------: | ----: | ------------: |
| 100 sessions x 3 turns x 2 artifacts | 16ms | 159ms | 1.1ms | 10ms | 185ms |
| 300 sessions x 900 turns, 12,000 artifacts | 14ms | 397ms | 1.2ms | 23ms | 435ms |
The 12k-artifact shape is the #4027 original scale. Against the
historical ~130s cold-start profile (~90s in the artifact phase), that
residual no longer exists on current main, and recovery scales
sub-linearly (30x the data, 2.5x the recover time). These are
single-machine synthetic-fixture measurements, not production
observations; the script is the deliverable so the baseline stays
reproducible on any machine.
Refs #4677.
Generated-by: GLM-5.3-Flash (ZCode)
perf(runtime-host): keep profile seed turns out of the typing simulation
FakeBackend paces non-steering responses as 9-char chunks with a 45ms
sleep per chunk, so the 283-char seed text spent ~1.4s per turn in that
typing simulation - 32 chunks of scripted pacing, not anything the
startup anchors measure. A ~25-char seed turn streams a few deltas over
the same durable path.
Seeding one session with two turns drops from 4.5s to 1.9s (same
machine, same shape otherwise), which compounds across fixture builds:
the 300-session fixture seeds in roughly 15 minutes instead of 40.
Generated-by: GLM-5.3-Flash (ZCode)
fix(runtime-host): attribute the artifact seeding phase once
The seed pass started the artifact-phase timer once before the upload
loop but added the cumulative elapsed time after every artifact, so with
N artifacts `PROFILE_DEBUG=1` reported N times the phase duration
(3 artifacts: artifacts 0.06s against a 0.04s session total, driving
`other` negative). Start the timer inside the loop per artifact, matching
the existing turnsMs pattern, so the phase is summed exactly once.
Generated-by: GLM-5.3-Flash (ZCode)
Merge remote-tracking branch 'upstream/main' into perf/startup-profile-script
# Conflicts:
# packages/runtime-host/package.json
Merge remote-tracking branch 'upstream/main' into perf/startup-profile-script
Merge remote-tracking branch 'upstream/main' into perf/startup-profile-script
# Conflicts:
# packages/runtime-host/package.json
---
packages/runtime-host/package.json | 1 +
packages/runtime-host/scripts/startup-profile.mjs | 313 ++++++++++++++++++++++
2 files changed, 314 insertions(+)
diff --git a/packages/runtime-host/package.json b/packages/runtime-host/package.json
index 747a077813..c55c2b5a47 100644
--- a/packages/runtime-host/package.json
+++ b/packages/runtime-host/package.json
@@ -26,6 +26,7 @@
"clean": "node ../../scripts/clean-paths.mjs dist tsconfig.tsbuildinfo",
"build": "tsc -p tsconfig.json",
"benchmark:session-copy": "node scripts/session-copy-cost-benchmark.mjs",
+ "benchmark:startup-profile": "node scripts/startup-profile.mjs",
"benchmark:hydration-e2e": "node scripts/hydration-e2e-benchmark.mjs",
"typecheck": "tsc -p tsconfig.json --noEmit",
"test:dist": "node --test \"dist/**/*.test.js\""
diff --git a/packages/runtime-host/scripts/startup-profile.mjs b/packages/runtime-host/scripts/startup-profile.mjs
new file mode 100644
index 0000000000..43a67e4e0d
--- /dev/null
+++ b/packages/runtime-host/scripts/startup-profile.mjs
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// Startup profile for the execution Runtime Host composition.
+//
+// Seeds a usage-shaped fixture (Sessions × Turns × artifacts), then reopens
+// the composition several times, timing each restart's anchors separately:
+//
+// open construct the execution composition (storage roots, stores)
+// recover the five-phase domain-module recovery
+// firstQuery the first `session.catalog.query` after recovery
+// close orderly composition shutdown
+//
+// Usage (from packages/runtime-host, after a workspace build):
+// node scripts/startup-profile.mjs
+// SESSIONS=300 TURNS=3 ARTIFACTS=4 ROUNDS=5 node scripts/startup-profile.mjs
+
+import { performance } from 'node:perf_hooks';
+import { createHash } from 'node:crypto';
+import { mkdtemp, rm, stat } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { randomUUID } from 'node:crypto';
+import { FakeBackend } from '@maka/runtime/test-only/fake-backend';
+import { SessionManager } from '@maka/runtime/session-manager';
+import { resolveStorageRoot, tryAcquireInteractiveRootOwner } from '@maka/storage/root-authority';
+import { createExecutionRuntimeHostComposition } from '../dist/server/execution-composition.js';
+
+const SESSIONS = Number(process.env.SESSIONS ?? 30);
+const TURNS = Number(process.env.TURNS ?? 3);
+const ARTIFACTS = Number(process.env.ARTIFACTS ?? 2);
+const ARTIFACT_BYTES = Number(process.env.ARTIFACT_BYTES ?? 2048);
+const ROUNDS = Number(process.env.ROUNDS ?? 3);
+
+const CONNECTION_ID = randomUUID();
+const operationContext = {
+ hostEpoch: 'startup-profile',
+ connectionId: 'startup-profile-client',
+ principal: 'local_os_user',
+ acquireResidency: () => ({ release() {} }),
+};
+
+const root = await mkdtemp(join(tmpdir(), 'maka-startup-profile-'));
+const capability = await resolveStorageRoot({ path: root, kind: 'interactive' });
+const owner = await tryAcquireInteractiveRootOwner(capability);
+if (!owner) throw new Error(`Could not acquire profile storage root: ${root}`);
+
+console.log(
+ `fixture target: ${SESSIONS} sessions × ${TURNS} turns × ` +
+ `${ARTIFACTS} artifacts(${ARTIFACT_BYTES}B), ${ROUNDS} restart rounds`,
+);
+
+// ── Seed pass ────────────────────────────────────────────────────────────────
+const seedOpenStart = performance.now();
+let seedComposition;
+try {
+ const seedOriginalRecover = SessionManager.prototype.recoverInterruptedSessionsStrict;
+ let seedManager;
+ SessionManager.prototype.recoverInterruptedSessionsStrict = async function (stores) {
+ seedManager = this;
+ return seedOriginalRecover.call(this, stores);
+ };
+
+ seedComposition = await createExecutionRuntimeHostComposition(
+ {
+ owner,
+ hostEpoch: operationContext.hostEpoch,
+ acquireResidency: () => ({ release() {} }),
+ retainUntilProcessExit: () => undefined,
+ requestDrain: () => undefined,
+ },
+ {},
+ { primaryBackendFactory: (context) => new FakeBackend(context) },
+ );
+ await seedComposition.recover();
+ SessionManager.prototype.recoverInterruptedSessionsStrict = seedOriginalRecover;
+ if (!seedManager) throw new Error('Seed composition did not construct a SessionManager');
+ await configureDefaultModel(seedComposition.handlers);
+
+ const artifactBytes = Buffer.from(
+ randomUUID()
+ .repeat(Math.ceil(ARTIFACT_BYTES / 36))
+ .slice(0, ARTIFACT_BYTES),
+ 'utf8',
+ );
+ const artifactSha = `sha256:${createHash('sha256').update(artifactBytes).digest('hex')}`;
+
+ for (let sessionIndex = 0; sessionIndex < SESSIONS; sessionIndex += 1) {
+ const sessionStart = performance.now();
+ let turnsMs = 0;
+ let artifactsMs = 0;
+ const created = await seedComposition.handlers['session.create'](
+ {
+ sessionId: randomUUID(),
+ workspace: { kind: 'host_path', path: root },
+ name: `startup-profile-${sessionIndex}`,
+ modelTarget: { kind: 'default' },
+ },
+ operationContext,
+ );
+ if (!created.ok) throw new Error(`session.create failed: ${JSON.stringify(created.error)}`);
+ const sessionId = created.result.id;
+
+ for (let turnIndex = 0; turnIndex < TURNS; turnIndex += 1) {
+ const turnStart = performance.now();
+ const turnId = `turn-${sessionIndex}-${turnIndex}`;
+ const started = await seedComposition.handlers['turn.start'](
+ {
+ sessionId,
+ turnId,
+ // FakeBackend paces non-steering responses as 9-char chunks with a
+ // 45ms typing sleep per chunk, so a long seed turn spends seconds in
+ // that simulation instead of the anchors this profile measures. A
+ // ~25-char turn still streams a few deltas on the same durable path.
+ content: { text: `startup profile turn ${sessionIndex}/${turnIndex}` },
+ },
+ operationContext,
+ );
+ if (!started.ok) throw new Error(`turn.start failed: ${JSON.stringify(started.error)}`);
+ await waitForCompletedTurn(seedManager, sessionId, turnId);
+ turnsMs += performance.now() - turnStart;
+ }
+
+ for (let artifactIndex = 0; artifactIndex < ARTIFACTS; artifactIndex += 1) {
+ const artifactStart = performance.now();
+ const uploadId = randomUUID();
+ const ingest = async (input) => {
+ const outcome = await seedComposition.handlers['artifact.ingest'](input, operationContext);
+ if (!outcome.ok) {
+ throw new Error(`artifact.ingest ${input.kind} failed: ${JSON.stringify(outcome.error)}`);
+ }
+ if (process.env.PROFILE_DEBUG) console.log(input.kind, JSON.stringify(outcome.result));
+ return outcome.result;
+ };
+ await ingest({
+ kind: 'begin',
+ sessionId,
+ uploadId,
+ name: `profile-${sessionIndex}-${artifactIndex}.bin`,
+ mimeType: 'application/octet-stream',
+ totalBytes: ARTIFACT_BYTES,
+ contentSha256: artifactSha,
+ });
+ // The wire limits each chunk's base64 payload; walk offsets from the
+ // accepted nextOffset instead of assuming one chunk fits.
+ let offset = 0;
+ const chunkBytes = 1536;
+ while (offset < ARTIFACT_BYTES) {
+ const slice = artifactBytes.subarray(offset, offset + chunkBytes);
+ const accepted = await ingest({
+ kind: 'chunk',
+ sessionId,
+ uploadId,
+ offset,
+ chunkBase64: slice.toString('base64'),
+ });
+ offset = accepted.nextOffset;
+ }
+ await ingest({ kind: 'commit', sessionId, uploadId });
+ artifactsMs += performance.now() - artifactStart;
+ }
+ if (process.env.PROFILE_DEBUG) {
+ console.log(
+ `session ${sessionIndex}: total=${((performance.now() - sessionStart) / 1000).toFixed(2)}s ` +
+ `turns=${(turnsMs / 1000).toFixed(2)}s artifacts=${(artifactsMs / 1000).toFixed(2)}s ` +
+ `other=${((performance.now() - sessionStart - turnsMs - artifactsMs) / 1000).toFixed(2)}s`,
+ );
+ }
+ }
+
+ const seeded = await seedManager.listSessions();
+ console.log(
+ `seed done in ${((performance.now() - seedOpenStart) / 1000).toFixed(1)}s: ` +
+ `${seeded.length} sessions on disk`,
+ );
+} finally {
+ await seedComposition?.close?.();
+}
+
+async function configureDefaultModel(handlers) {
+ const call = async (operation, input) => {
+ const outcome = await handlers[operation](input, operationContext);
+ if (!outcome.ok) throw new Error(`${operation} failed: ${JSON.stringify(outcome.error)}`);
+ return outcome.result;
+ };
+ const catalog = await call('connection.catalog.query', { kind: 'start' });
+ const created = await call('connection.catalog.create', {
+ expectedCatalogRevision: catalog.revision,
+ connection: {
+ slug: 'startup-profile',
+ name: 'Startup profile',
+ providerType: 'custom',
+ defaultApiProtocol: 'openai-chat',
+ baseUrl: 'https://startup-profile.invalid/v1',
+ enabled: true,
+ enabledModelIds: ['profile-test-model'],
+ },
+ });
+ if (created.kind !== 'committed') throw new Error('Profile connection must commit');
+ await call('credential.vault.set', {
+ locator: {
+ scope: 'connection',
+ connectionId: created.connection.connectionId,
+ kind: 'api_key',
+ },
+ expected: null,
+ secret: 'startup-profile-key',
+ });
+ await call('connection.catalog.set-default-target', {
+ expectedCatalogRevision: created.catalogRevision,
+ target: { connectionId: created.connection.connectionId, modelId: 'profile-test-model' },
+ });
+}
+
+// ── Restart rounds ───────────────────────────────────────────────────────────
+const rounds = [];
+for (let round = 0; round < ROUNDS; round += 1) {
+ rounds.push(await restartRound(round));
+}
+console.table(
+ rounds.map((round) => ({
+ round: round.round,
+ openMs: round.openMs.toFixed(0),
+ recoverMs: round.recoverMs.toFixed(0),
+ firstQueryMs: round.firstQueryMs.toFixed(1),
+ closeMs: round.closeMs.toFixed(0),
+ totalMs: round.totalMs.toFixed(0),
+ })),
+);
+const med = (values) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)];
+console.log(
+ `median: open=${med(rounds.map((r) => r.openMs)).toFixed(0)}ms ` +
+ `recover=${med(rounds.map((r) => r.recoverMs)).toFixed(0)}ms ` +
+ `firstQuery=${med(rounds.map((r) => r.firstQueryMs)).toFixed(1)}ms ` +
+ `close=${med(rounds.map((r) => r.closeMs)).toFixed(0)}ms ` +
+ `total=${med(rounds.map((r) => r.totalMs)).toFixed(0)}ms`,
+);
+const rootInfo = await stat(root);
+console.log(`storage root: ${root} (${(rootInfo.size / 1024).toFixed(1)} KiB top-level)`);
+await rm(root, { recursive: true, force: true });
+
+async function restartRound(round) {
+ const openStart = performance.now();
+ const composition = await createExecutionRuntimeHostComposition(
+ {
+ owner,
+ hostEpoch: `${operationContext.hostEpoch}-${round}`,
+ acquireResidency: () => ({ release() {} }),
+ retainUntilProcessExit: () => undefined,
+ requestDrain: () => undefined,
+ },
+ {},
+ { primaryBackendFactory: (context) => new FakeBackend(context) },
+ );
+ const openMs = performance.now() - openStart;
+
+ const recoverStart = performance.now();
+ await composition.recover();
+ const recoverMs = performance.now() - recoverStart;
+
+ const queryStart = performance.now();
+ const listed = await composition.handlers['session.catalog.query'](
+ { kind: 'list_start' },
+ operationContext,
+ );
+ const firstQueryMs = performance.now() - queryStart;
+ if (!listed.ok) throw new Error(`catalog query failed: ${JSON.stringify(listed.error)}`);
+
+ const closeStart = performance.now();
+ await composition.close?.();
+ const closeMs = performance.now() - closeStart;
+
+ return {
+ round,
+ openMs,
+ recoverMs,
+ firstQueryMs,
+ closeMs,
+ totalMs: performance.now() - openStart,
+ };
+}
+
+async function waitForCompletedTurn(manager, sessionId, turnId) {
+ const deadline = performance.now() + 15_000;
+ while (performance.now() < deadline) {
+ const messages = await manager.getMessages(sessionId);
+ const state = messages.findLast(
+ (message) => message.type === 'turn_state' && message.turnId === turnId,
+ );
+ if (state?.status === 'completed') return;
+ if (state?.status === 'failed' || state?.status === 'aborted') {
+ throw new Error(`Profile turn ${turnId} ended as ${state.status}`);
+ }
+ await new Promise((resolve) => setTimeout(resolve, 5));
+ }
+ throw new Error(`Timed out waiting for profile turn ${turnId}`);
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org