You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/09/11 12:58:23 UTC
(skywalking-ai-sessionizer) branch main updated: Remove a scenario session once it is sent, by the policy its build chose
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/skywalking-ai-sessionizer.git
The following commit(s) were added to refs/heads/main by this push:
new 1150dcd Remove a scenario session once it is sent, by the policy its build chose
1150dcd is described below
commit 1150dcd9d3fbf6a513de4cb26d83c9153684fd0f
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Fri Sep 11 20:55:58 2026 +0800
Remove a scenario session once it is sent, by the policy its build chose
A scenario feed writes sessions until it is stopped, and nothing removed
them, so a demo's storage root grew without end. Removal is a policy of
the scenario, never an option of the product. asz scenario build takes
--remove immediately, the default, or --remove with a duration, and writes
it into a marker for each session after every other file of it. asz
collect and asz server remove a marked session at the end of a pass, once
every landed file, round and metrics request of it was sent to the
receiver they send to and none was rejected. A session no build marked,
which is every real Claude Code session, is never removed.
A crash must never send half a session. The source files go first, so
nothing lands them again. The session and chain directories are each
renamed into _removed/ in one call before they are deleted, so nothing
parses what is left of them. A pipeline over a scenario root holds a root
lock, so a second one waits. Parse now takes the chain lock before it
reads the index. push.state records the endpoint it sent to and the files
a receiver rejected, and an HTTP answer counts as a success only when its
body is empty or protobuf.
---
CLAUDE.md | 7 +-
cmd/asz/pipeline_test.go | 426 ++++++-
cmd/asz/push.go | 7 +-
cmd/asz/refresh.go | 195 ++-
cmd/asz/scenario.go | 37 +-
cmd/asz/scenario_test.go | 52 +
cmd/asz/view.go | 18 +-
docs/en/changes/changes.md | 29 +
docs/en/formats/storage-root.md | 93 +-
docs/en/guides/scenario.md | 88 +-
docs/en/setup/command-line.md | 80 +-
docs/en/setup/export-otlp.md | 41 +-
internal/adapters/claudecode/covered.go | 135 +++
internal/adapters/claudecode/covered_test.go | 210 ++++
internal/adapters/claudecode/tail.go | 4 +-
internal/adapters/claudecodechanges/covered.go | 55 +
.../adapters/claudecodechanges/covered_test.go | 108 ++
internal/export/otlp/client.go | 38 +-
internal/export/otlp/client_test.go | 139 +++
internal/export/otlp/otlptest/receiver.go | 69 +-
internal/export/otlp/push.go | 98 +-
internal/export/otlp/sent.go | 150 +++
internal/export/otlp/sent_test.go | 309 +++++
internal/metrics/metrics.go | 3 +-
internal/metrics/session.go | 172 +++
internal/metrics/session_test.go | 223 ++++
internal/parse/parse.go | 100 +-
internal/parse/parse_test.go | 174 +++
internal/scenario/build.go | 24 +-
internal/scenario/expect/expect.go | 6 +
internal/scenario/marker.go | 282 +++++
internal/scenario/marker_test.go | 348 ++++++
internal/scenario/plan.go | 4 +
internal/scenario/remove/remove.go | 1282 ++++++++++++++++++++
internal/scenario/remove/remove_test.go | 861 +++++++++++++
internal/scenario/run/remove.go | 696 +++++++++++
internal/scenario/run/run.go | 45 +-
internal/scenario/write_claudecode.go | 70 +-
internal/storage/cursor.go | 3 +
internal/storage/lock.go | 42 +
internal/storage/lock_test.go | 62 +
internal/storage/removed.go | 273 +++++
internal/storage/removed_test.go | 410 +++++++
internal/storage/zone.go | 5 +
tests/chain/remove_test.go | 602 +++++++++
tests/scenarios/removed-after-sent.expect.yaml | 36 +
tests/scenarios/removed-after-sent.yaml | 46 +
47 files changed, 8025 insertions(+), 132 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index cc1543a..88f57c8 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -68,8 +68,9 @@ internal/repack/ re-cuts landed files into a new root under anoth
internal/adapters/claudecode/ the claude-code-local adapter
internal/adapters/claudecodeotlp/ the claude-code-otlp adapter: a receiver for the runtime's own exporter, metrics into the spool
internal/adapters/mock/ the mock dialect: Session Data a scenario writes directly
-internal/scenario/ scenarios: the model, the clock, and the two writers (collector side)
+internal/scenario/ scenarios: the model, the clock, the two writers, and each session's marker (collector side)
internal/scenario/expect/ expectation files evaluated over a root (server side)
+internal/scenario/remove/ removes a sent scenario session by its marker's policy (collector side)
internal/scenario/run/ the runner that wires build, collect, parse and check; command and tests only
tests/scenarios/ the assembly tests, one scenario and one expectation file each
internal/config/ YAML configuration
@@ -157,6 +158,10 @@ Every source file carries the Apache-2.0 header; `make license-fix` inserts miss
## Invariants worth knowing before changing collector code
- **Landed files are write-once.** temp → fsync → rename → `chmod 0444`. Never appended.
+- **asz never removes a real session.** Only a session a scenario build marked is removed, by the
+ policy in its marker, and only once all of it was sent to the one receiver the pipeline sends to.
+ The product configuration has no removal setting. A removal renames each directory away before it
+ deletes it, so no reader sees half a session.
- **A landed file is cut once.** A round addresses records by `{seq, row}` and binds to file
digests, so a referenced file is never re-cut. A new budget applies to new files; an existing
root is brought under it with `asz repack` into a new root, where the chains are built again.
diff --git a/cmd/asz/pipeline_test.go b/cmd/asz/pipeline_test.go
index c26b974..d2fa49c 100644
--- a/cmd/asz/pipeline_test.go
+++ b/cmd/asz/pipeline_test.go
@@ -18,6 +18,8 @@
package main
import (
+ "io"
+ "os"
"path/filepath"
"strings"
"testing"
@@ -29,6 +31,7 @@ import (
"github.com/apache/skywalking-ai-sessionizer/internal/config"
"github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
"github.com/apache/skywalking-ai-sessionizer/internal/export/otlp/otlptest"
+ "github.com/apache/skywalking-ai-sessionizer/internal/metrics"
"github.com/apache/skywalking-ai-sessionizer/internal/scenario"
"github.com/apache/skywalking-ai-sessionizer/internal/storage"
"github.com/apache/skywalking-ai-sessionizer/internal/view"
@@ -260,6 +263,10 @@ func testRefresher(t *testing.T, zone *storage.Zone, root string) *refresher {
if ref == nil {
t.Fatal("no refresher for a root that has a source beside it")
}
+ // A scenario root's lock is held for the life of a process. A test ends
+ // long before its process does, and the next one may put another
+ // refresher on the same root.
+ t.Cleanup(ref.close)
return ref
}
@@ -376,12 +383,18 @@ func TestAnotherBuilderHoldingTheChainIsNotAFailure(t *testing.T) {
}
// A single pass has no next pass to carry it to, so it must say so
- // rather than exit clean with data unparsed.
+ // rather than exit clean with data unparsed. It is a second pipeline
+ // over a scenario root, so the first lets go of the root before it runs,
+ // or it would wait for the root and fail for that reason instead.
+ ref.close()
final := testRefresher(t, zone, root)
final.last = true
if err := final.pass(); err == nil {
t.Fatal("a single pass exited clean while another builder held a session it had landed for")
+ } else if strings.Contains(err.Error(), "scenario root") {
+ t.Fatalf("the single pass failed on the scenario root, not on the chain: %v", err)
}
+ final.close()
// The other builder is done. The next pass must pick the session up
// again, even though its source has not grown since.
@@ -465,3 +478,414 @@ func TestOnlyOnePassSendsAtATime(t *testing.T) {
t.Fatal("nothing was sent once the export state was free")
}
}
+
+// scenarioRoot builds one claude-code session into a fresh root. Claude
+// Code's own directory is a place of the test's, so no guard depends on the
+// machine the test runs on.
+func scenarioRoot(t *testing.T, name string) (string, *scenario.Built) {
+ t.Helper()
+ t.Setenv("CLAUDE_CONFIG_DIR", t.TempDir())
+ root := t.TempDir()
+ sc, err := scenario.Load(filepath.Join("..", "..", "tests", "scenarios", name))
+ if err != nil {
+ t.Fatal(err)
+ }
+ b, err := scenario.Build(sc, scenario.FormatClaudeCode, root, scenario.Options{At: time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ return root, b
+}
+
+// bothAdapters is what a build's configuration enables: both local
+// adapters, the plugin's reading under the build's source unless another
+// directory is given, and metrics derived with no look-back.
+func bothAdapters(root, changes string) []config.Adapter {
+ source := filepath.Join(root, "_source")
+ if changes == "" {
+ changes = filepath.Join(source, "plugins", "data")
+ }
+ return []config.Adapter{
+ {Name: config.AdapterClaudeCodeLocal, Enabled: true, SourceRoot: source, Metrics: true, MetricsLookback: "none",
+ Collector: config.Collector{Mode: config.ModeOnce, Interval: time.Second, MaxDeltaBytes: 1 << 20}},
+ {Name: config.AdapterClaudeCodeChanges, Enabled: true, SourceRoot: changes,
+ Collector: config.Collector{Mode: config.ModeOnce, Interval: time.Second, MaxDeltaBytes: 1 << 20}},
+ }
+}
+
+// pipelineOver is the pipeline asz collect -once runs over a root.
+func pipelineOver(t *testing.T, root string, ads []config.Adapter) *refresher {
+ t.Helper()
+ zone := storage.NewZone(root)
+ ref, err := newRefresher(view.New(zone, nil), zone, ads, 2<<20, true)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(ref.close)
+ return ref
+}
+
+func receiver(t *testing.T) *otlptest.Receiver {
+ t.Helper()
+ rcv, err := otlptest.Start()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(rcv.Close)
+ return rcv
+}
+
+// sendingTo gives the pipeline a pusher that records the receiver it sends
+// to, as newPusher builds one.
+func sendingTo(t *testing.T, ref *refresher, rcv *otlptest.Receiver) {
+ t.Helper()
+ o := rcv.Options(otlp.ProtocolGRPC)
+ client, err := otlp.NewClient(o)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { client.Close() })
+ ref.pusher = &otlp.Pusher{
+ Zone: ref.zone, Client: client, Endpoint: otlp.EndpointOf(o.Protocol, o.Endpoint, o.TLS), Version: "test",
+ ServiceName: "Removal Test", InstanceID: "removal-test", Layer: "AI_AGENT", BatchBytes: 8 << 20,
+ MetricsService: "Removal Test",
+ }
+ if err := ref.pusher.Prepare(); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// stderrOf runs fn and returns what it wrote to standard error, where a pass
+// says why it kept a session.
+func stderrOf(t *testing.T, fn func()) string {
+ t.Helper()
+ return written(t, &os.Stderr, fn)
+}
+
+// stdoutOf runs fn and returns what it wrote to standard output, where the
+// pipeline says what it will do.
+func stdoutOf(t *testing.T, fn func()) string {
+ t.Helper()
+ return written(t, &os.Stdout, fn)
+}
+
+// written runs fn with *stream sent into a pipe, and returns what fn wrote.
+func written(t *testing.T, stream **os.File, fn func()) string {
+ t.Helper()
+ r, w, err := os.Pipe()
+ if err != nil {
+ t.Fatal(err)
+ }
+ out := make(chan string)
+ go func() {
+ data, _ := io.ReadAll(r)
+ out <- string(data)
+ }()
+ old := *stream
+ *stream = w
+ func() {
+ defer func() { *stream = old; _ = w.Close() }()
+ fn()
+ }()
+ return <-out
+}
+
+func pathExists(p string) bool {
+ _, err := os.Lstat(p)
+ return err == nil
+}
+
+// TestOnePassSendsAndRemovesAScenarioSession. Over a root a claude-code
+// build wrote, the pass that sends a session whole also removes it, and the
+// page stops listing it. The next pass has nothing to send or remove.
+func TestOnePassSendsAndRemovesAScenarioSession(t *testing.T) {
+ root, b := scenarioRoot(t, "workspace-changes.yaml")
+ rcv := receiver(t)
+ ref := pipelineOver(t, root, bothAdapters(root, ""))
+ sendingTo(t, ref, rcv)
+ if err := ref.pass(); err != nil {
+ t.Fatal(err)
+ }
+ if len(rcv.Requests()) == 0 {
+ t.Fatal("the pass sent nothing")
+ }
+ for _, p := range []string{b.Marker, filepath.Join(root, b.Session), filepath.Join(root, "_conversations", b.Session)} {
+ if pathExists(p) {
+ t.Errorf("%s is still there after the pass that sent all of it", p)
+ }
+ }
+ ids, err := ref.srv.List()
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, id := range ids {
+ if id == b.Session {
+ t.Fatal("the page still lists the removed conversation")
+ }
+ }
+ rcv.Reset()
+ if err := ref.pass(); err != nil {
+ t.Fatal(err)
+ }
+ if n := len(rcv.Requests()) + len(rcv.MetricsRequests()); n != 0 {
+ t.Fatalf("the next pass sent %d request(s)", n)
+ }
+ if pathExists(filepath.Join(root, b.Session)) {
+ t.Fatal("the next pass made the removed session again")
+ }
+}
+
+// TestAScenarioSessionIsKeptWithoutAnEndpoint. With nothing sent, nothing
+// can be proved sent, so nothing is removed. The pass says why, once, and it
+// is not a failure: the product has no removal setting that could be wrong.
+func TestAScenarioSessionIsKeptWithoutAnEndpoint(t *testing.T) {
+ root, b := scenarioRoot(t, "workspace-changes.yaml")
+ ref := pipelineOver(t, root, bothAdapters(root, ""))
+ var err error
+ said := stderrOf(t, func() { err = ref.pass() })
+ if err != nil {
+ t.Fatalf("a pass with no endpoint failed: %v", err)
+ }
+ if !strings.Contains(said, "kept: every marked session: no export endpoint") {
+ t.Fatalf("the pass did not say why the session is kept:\n%s", said)
+ }
+ if !pathExists(b.Marker) || !pathExists(filepath.Join(root, b.Session)) {
+ t.Fatal("a session was removed with no endpoint to send it to")
+ }
+ if said := stderrOf(t, func() { _ = ref.pass() }); strings.Contains(said, "kept:") {
+ t.Fatalf("the reason was said a second time:\n%s", said)
+ }
+}
+
+// TestAScenarioSessionIsKeptWhenTheChangesAdapterReadsElsewhere. The build
+// wrote plugin output under its own source. A changes adapter that reads any
+// other directory never lands it, so it is never sent, and the session stays.
+func TestAScenarioSessionIsKeptWhenTheChangesAdapterReadsElsewhere(t *testing.T) {
+ root, b := scenarioRoot(t, "workspace-changes.yaml")
+ rcv := receiver(t)
+ ref := pipelineOver(t, root, bothAdapters(root, t.TempDir()))
+ sendingTo(t, ref, rcv)
+ var err error
+ said := stderrOf(t, func() { err = ref.pass() })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(said, "kept: every marked session: the changes adapter reads") {
+ t.Fatalf("the pass did not say why the session is kept:\n%s", said)
+ }
+ if !pathExists(b.Marker) || !pathExists(filepath.Join(root, b.Session)) {
+ t.Fatal("a session was removed while its plugin output could never be landed")
+ }
+}
+
+// TestTheScenarioLineSaysWhyNothingIsRemoved. At the start, over a scenario
+// root, the pipeline says what the removal will do. It asks the check each
+// removal pass makes before it looks at any session, so it never promises a
+// removal the configuration prevents. The line printed when a pass takes the
+// lock says the same.
+func TestTheScenarioLineSaysWhyNothingIsRemoved(t *testing.T) {
+ root, _ := scenarioRoot(t, "workspace-changes.yaml")
+ rcv := receiver(t)
+ cfg := &config.Config{}
+ cfg.Export.OTLP.Endpoint = rcv.Options(otlp.ProtocolGRPC).Endpoint
+ for _, c := range []struct {
+ name string
+ ads []config.Adapter
+ send bool
+ // metricsOff and logsOff are what export.otlp.metrics: false and
+ // export.otlp.logs: false do to the pusher.
+ metricsOff bool
+ logsOff bool
+ want string
+ }{
+ {name: "no endpoint", ads: bothAdapters(root, ""),
+ want: "scenario : no session is removed: no export endpoint; nothing is sent, so nothing is removed\n"},
+ {name: "metrics off", ads: bothAdapters(root, ""), send: true, metricsOff: true,
+ want: "scenario : no session is removed: export.otlp.metrics is off;"},
+ {name: "logs off", ads: bothAdapters(root, ""), send: true, logsOff: true,
+ want: "scenario : no session is removed: export.otlp.logs is off;"},
+ {name: "the changes adapter elsewhere", ads: bothAdapters(root, t.TempDir()), send: true,
+ want: "scenario : no session is removed: the changes adapter reads "},
+ {name: "no claude-code-local", ads: bothAdapters(root, "")[1:], send: true,
+ want: "scenario : no session is removed: claude-code-local is not enabled"},
+ {name: "nothing prevents it", ads: bothAdapters(root, ""), send: true,
+ want: "scenario : a session a build marked is removed by its policy once all of it is sent\n"},
+ } {
+ t.Run(c.name, func(t *testing.T) {
+ ref := pipelineOver(t, root, c.ads)
+ if c.send {
+ sendingTo(t, ref, rcv)
+ ref.pusher.NoMetrics = c.metricsOff
+ ref.pusher.NoLogs = c.logsOff
+ }
+ said := stdoutOf(t, func() { printPipeline(ref, cfg) })
+ if !strings.Contains(said, c.want) {
+ t.Fatalf("the start-up lines do not say %q:\n%s", c.want, said)
+ }
+ })
+ }
+
+ // A root no build wrote has no such line.
+ plain := t.TempDir()
+ said := stdoutOf(t, func() { printPipeline(pipelineOver(t, plain, bothAdapters(plain, "")), cfg) })
+ if strings.Contains(said, "scenario :") {
+ t.Fatalf("a root with no %s has a scenario line:\n%s", storage.ScenarioDir, said)
+ }
+
+ // The lock's line ends with the same words as the start-up line.
+ ref := pipelineOver(t, root, bothAdapters(root, ""))
+ said = stdoutOf(t, func() { _ = stderrOf(t, func() { _ = ref.pass() }) })
+ want := "scenario : this pipeline holds " + filepath.Join(root, storage.ScenarioDir, ".lock") +
+ "; no session is removed: no export endpoint; nothing is sent, so nothing is removed\n"
+ if !strings.Contains(said, want) {
+ t.Fatalf("the lock's line is not %q:\n%s", want, said)
+ }
+}
+
+// TestASecondPipelineWaitsForTheScenarioRoot. Two pipelines over one
+// scenario root would race a removal: one could land again what the other
+// just removed. So the second does nothing while the first holds the root.
+func TestASecondPipelineWaitsForTheScenarioRoot(t *testing.T) {
+ root, b := scenarioRoot(t, "assembly.yaml")
+ held, err := storage.LockScenario(root)
+ if err != nil {
+ t.Fatal(err)
+ }
+ zone := storage.NewZone(root)
+ watching := testRefresher(t, zone, root)
+ if err := watching.pass(); err != nil {
+ t.Fatalf("a watching pass over a root another pipeline holds failed: %v", err)
+ }
+ if pathExists(filepath.Join(root, b.Session)) || pathExists(filepath.Join(root, "_conversations")) {
+ t.Fatal("a pass over a root another pipeline holds landed or parsed")
+ }
+ single := testRefresher(t, zone, root)
+ single.last, single.scenarioWait = true, 200*time.Millisecond
+ if err := single.pass(); err == nil || !strings.Contains(err.Error(), "another pipeline holds this scenario root") {
+ t.Fatalf("a single pass over a held root gave %v", err)
+ }
+ if err := held.Unlock(); err != nil {
+ t.Fatal(err)
+ }
+ if err := watching.pass(); err != nil {
+ t.Fatal(err)
+ }
+ rounds, err := filepath.Glob(filepath.Join(root, "_conversations", b.Session, "rounds", "*"))
+ if err != nil || len(rounds) == 0 {
+ t.Fatalf("once the root was released the pass wrote %d round(s): %v", len(rounds), err)
+ }
+}
+
+// TestTheDeriverSeesTheChangesAdaptersSessions. After the first pass, the
+// derivation takes only the sessions a pass moved. A session that only the
+// changes adapter moved has to be among them. Otherwise its new files wait
+// for a pass that moves nothing at all, which never comes while a feed runs.
+func TestTheDeriverSeesTheChangesAdaptersSessions(t *testing.T) {
+ t.Setenv("CLAUDE_CONFIG_DIR", t.TempDir())
+ root := t.TempDir()
+ at := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+ growth, err := scenario.Load(filepath.Join("..", "..", "tests", "scenarios", "growth.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ withChanges, err := scenario.Load(filepath.Join("..", "..", "tests", "scenarios", "workspace-changes.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := scenario.Build(growth, scenario.FormatClaudeCode, root, scenario.Options{At: at, Through: "turn1"}); err != nil {
+ t.Fatal(err)
+ }
+ b, err := scenario.Build(withChanges, scenario.FormatClaudeCode, root, scenario.Options{At: at.Add(time.Hour)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ // The plugin's output is not there on the first pass.
+ plugin := filepath.Join(root, "_source", filepath.FromSlash(scenario.PluginOutputDir), "..")
+ hidden := filepath.Join(root, "_source", "plugins", "hidden")
+ if err := os.Rename(plugin, hidden); err != nil {
+ t.Fatal(err)
+ }
+ ref := pipelineOver(t, root, bothAdapters(root, ""))
+ if err := ref.pass(); err != nil {
+ t.Fatal(err)
+ }
+ // Now it is, and the local adapter moves only the other session.
+ if err := os.Rename(hidden, plugin); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := scenario.Build(growth, scenario.FormatClaudeCode, root, scenario.Options{At: at}); err != nil {
+ t.Fatal(err)
+ }
+ if err := ref.pass(); err != nil {
+ t.Fatal(err)
+ }
+ zone := storage.NewZone(root)
+ files, err := storage.LandedFiles(zone, b.Session)
+ if err != nil {
+ t.Fatal(err)
+ }
+ derived, err := metrics.LoadDerived(zone)
+ if err != nil {
+ t.Fatal(err)
+ }
+ saw := 0
+ for _, lf := range files {
+ if !strings.HasPrefix(filepath.Base(lf.Path), "changes-") {
+ continue
+ }
+ saw++
+ rel, _ := filepath.Rel(root, lf.Path)
+ digest, err := storage.FileDigest(lf.Path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !derived.Has(filepath.ToSlash(rel), digest) {
+ t.Errorf("%s, which only the changes adapter landed, was not derived", rel)
+ }
+ }
+ if saw == 0 {
+ t.Fatal("the second pass landed no plugin output")
+ }
+}
+
+// TestNoListingSeesRemoved. A removal renames a directory under _removed
+// before it deletes it. Until it is deleted, nothing that lists sessions or
+// chains may see it, or it would be parsed, sent or shown again.
+func TestNoListingSeesRemoved(t *testing.T) {
+ first, b := scenarioRoot(t, "assembly.yaml")
+ if err := pipelineOver(t, first, bothAdapters(first, "")).pass(); err != nil {
+ t.Fatal(err)
+ }
+ root := t.TempDir()
+ if err := os.MkdirAll(filepath.Join(root, "_source", "plugins", "data"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ removed := filepath.Join(root, storage.RemovedDir)
+ if err := os.MkdirAll(removed, 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Rename(filepath.Join(first, b.Session), filepath.Join(removed, b.Session)); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Rename(filepath.Join(first, "_conversations", b.Session), filepath.Join(removed, b.Session+".chain")); err != nil {
+ t.Fatal(err)
+ }
+ rcv := receiver(t)
+ ref := pipelineOver(t, root, bothAdapters(root, ""))
+ sendingTo(t, ref, rcv)
+ if err := ref.pass(); err != nil {
+ t.Fatalf("a pass over a root with a removal in progress failed: %v", err)
+ }
+ if rounds, _ := filepath.Glob(filepath.Join(root, "_conversations", "*", "rounds", "*")); len(rounds) != 0 {
+ t.Fatalf("the pass wrote %d round(s) from what was removed", len(rounds))
+ }
+ if n := len(rcv.Requests()) + len(rcv.MetricsRequests()); n != 0 {
+ t.Fatalf("the pass sent %d request(s) from what was removed", n)
+ }
+ if chains, _, _, err := verifyChains(root, ""); err != nil || chains != 0 {
+ t.Fatalf("asz verify found %d chain(s): %v", chains, err)
+ }
+ if ids, err := view.New(storage.NewZone(root), nil).List(); err != nil || len(ids) != 0 {
+ t.Fatalf("the page lists %v: %v", ids, err)
+ }
+}
diff --git a/cmd/asz/push.go b/cmd/asz/push.go
index 422db55..877fc43 100644
--- a/cmd/asz/push.go
+++ b/cmd/asz/push.go
@@ -127,8 +127,11 @@ func newPusher(cfg *config.Config, zoneRoot string) (*otlp.Pusher, func(), error
// each session, read off its landed headers: one service per kind of
// agent, which is how a receiver lists them.
p := &otlp.Pusher{
- Zone: storage.NewZone(zoneRoot),
- Client: client,
+ Zone: storage.NewZone(zoneRoot),
+ Client: client,
+ // Recorded beside what is sent, so a scenario root removes a session
+ // only once it is sent to the receiver this pipeline sends to.
+ Endpoint: otlp.EndpointOf(o.Protocol, o.Endpoint, o.TLS),
Version: version,
ServiceName: o.ServiceName,
Runtimes: map[string]string{claudecode.Name: claudecode.RuntimeName, claudecodechanges.Name: claudecodechanges.RuntimeName, mock.Name: mock.RuntimeName},
diff --git a/cmd/asz/refresh.go b/cmd/asz/refresh.go
index f8b7476..d164ab6 100644
--- a/cmd/asz/refresh.go
+++ b/cmd/asz/refresh.go
@@ -20,7 +20,9 @@ package main
import (
"errors"
"fmt"
+ "io/fs"
"os"
+ "path/filepath"
"sort"
"strings"
"time"
@@ -30,6 +32,7 @@ import (
"github.com/apache/skywalking-ai-sessionizer/internal/config"
"github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
"github.com/apache/skywalking-ai-sessionizer/internal/metrics"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario/remove"
"github.com/apache/skywalking-ai-sessionizer/internal/storage"
"github.com/apache/skywalking-ai-sessionizer/internal/view"
)
@@ -86,6 +89,23 @@ type refresher struct {
// only at what moved, and a session whose source has stopped growing
// never moves again. So they are carried until one pass gets the lock.
retry map[string]bool
+
+ // remover removes the sessions a scenario build marked, once all of
+ // each is sent. It exists when claude-code-local is enabled, and it runs
+ // only while this pipeline holds the scenario lock. A root no build wrote
+ // has no marker, so nothing there is ever removed.
+ remover *remove.Remover
+ // scenarioLock is held for the life of the process once the root is a
+ // scenario root. scenarioWait is how long a single pass waits for it.
+ scenarioLock *storage.SessionLock
+ scenarioWait time.Duration
+ // waitingSaid says a pass already reported that another pipeline holds
+ // the scenario root, so later passes wait quietly.
+ waitingSaid bool
+ // derive holds sessions the removal found not derived yet. The next
+ // derivation takes them beside what moved, because a session that no
+ // longer lands anything would otherwise never be derived again.
+ derive map[string]bool
}
// newRefresher wires the local adapters into one pipeline, or returns nil
@@ -153,9 +173,115 @@ func newRefresher(srv *view.Server, zone *storage.Zone, ads []config.Adapter, ma
if mode == config.ModeWatch {
r.base.IntervalMS = r.interval.Milliseconds()
}
+ // A build writes a session under claude-code-local's source, so only a
+ // pipeline that collects from there can remove one.
+ if r.col != nil {
+ r.remover = &remove.Remover{
+ Zone: zone, Source: r.colSource, Local: r.col, LocalMatch: r.match,
+ Changes: r.changes, ChangesRoot: r.changesSource, ChangesMatch: r.cmatch,
+ Derives: r.deriver != nil,
+ }
+ }
+ r.scenarioWait = exportWait
return r, nil
}
+// holdScenario takes the scenario lock when the root is a scenario root, and
+// reports whether this pass may go on. A root is one when a claude-code
+// build created _scenario in it. The check runs on every pass, so a feed
+// started after the pipeline is covered from its first session.
+//
+// A second pipeline over the same root does nothing at all, rather than run
+// without removing. Its collector could make an emptied session directory
+// again and save cursors there, and its parser could publish over evidence
+// that is gone.
+func (r *refresher) holdScenario() (bool, error) {
+ if r.scenarioLock != nil {
+ return true, nil
+ }
+ root := r.zone.Root()
+ fi, err := os.Stat(filepath.Join(root, storage.ScenarioDir))
+ switch {
+ case errors.Is(err, fs.ErrNotExist):
+ return true, nil
+ case err != nil:
+ return false, err
+ case !fi.IsDir():
+ return true, nil
+ }
+ lockPath := filepath.Join(root, storage.ScenarioDir, ".lock")
+ var lock *storage.SessionLock
+ if r.last {
+ // A single pass has to do its work or say it could not.
+ lock, err = storage.LockScenarioWait(root, r.scenarioWait)
+ if errors.Is(err, storage.ErrScenarioBusy) {
+ err = fmt.Errorf("another pipeline holds this scenario root (%s) and did not release it within %s; nothing was done", lockPath, r.scenarioWait)
+ st := r.base
+ st.LastError = err.Error()
+ r.setStatus(st)
+ return false, err
+ }
+ } else {
+ lock, err = storage.LockScenario(root)
+ if errors.Is(err, storage.ErrScenarioBusy) {
+ if !r.waitingSaid {
+ msg := fmt.Sprintf("another pipeline holds %s; this one does nothing until it is released", lockPath)
+ fmt.Printf("waiting : %s\n", msg)
+ st := r.base
+ if r.srv != nil {
+ st.LastRefresh = r.srv.Status().LastRefresh
+ }
+ st.LastError = msg
+ r.setStatus(st)
+ r.waitingSaid = true
+ }
+ return false, nil
+ }
+ }
+ if err != nil {
+ return false, err
+ }
+ r.scenarioLock, r.waitingSaid = lock, false
+ fmt.Printf("scenario : this pipeline holds %s; %s\n", lockPath, r.removalSays())
+ return true, nil
+}
+
+// close releases the scenario lock. The commands never call it, because the
+// lock lives as long as the process, and the system releases it when the
+// process ends. Tests call it, to put a second pipeline on the same root.
+func (r *refresher) close() {
+ if r.scenarioLock != nil {
+ _ = r.scenarioLock.Unlock()
+ r.scenarioLock = nil
+ }
+}
+
+// removeInput is what a removal needs to know about this pipeline that the
+// root does not say: where it sends, what it sends, and what it is holding
+// back to parse again.
+func (r *refresher) removeInput() remove.Input {
+ in := remove.Input{Busy: r.retry}
+ if r.pusher != nil {
+ in.Endpoint, in.SendLogs, in.SendMetrics = r.pusher.Endpoint, !r.pusher.NoLogs, !r.pusher.NoMetrics
+ }
+ return in
+}
+
+// removalSays is what a scenario : line says about removal. It asks the
+// check each removal pass makes before it looks at any session. Before, the
+// line promised a removal whenever there was an endpoint, even with metrics
+// sending off or the changes adapter reading another directory, where every
+// session is kept.
+func (r *refresher) removalSays() string {
+ if r.remover == nil {
+ return "no session is removed: claude-code-local is not enabled, and a build writes every session under its source directory"
+ }
+ if reason := r.remover.Blocked(r.removeInput()); reason != "" {
+ return "no session is removed: " + reason
+ }
+ return "a session a build marked is removed by its policy once all of it is sent"
+}
+
// present reports whether a source directory is there to be read.
//
// optional says what a missing directory means. For the plugin's records it
@@ -186,6 +312,11 @@ func present(dir string, optional bool) (bool, error) {
// than leave a backfill half done without saying so. A watching pass logs
// the same and goes round again.
func (r *refresher) pass() error {
+ // Before anything else. Over a scenario root, a pass that cannot hold
+ // the root lands, parses and sends nothing.
+ if held, err := r.holdScenario(); !held {
+ return err
+ }
start := time.Now()
first := r.full
@@ -268,10 +399,13 @@ func (r *refresher) pass() error {
}
}
if r.deriver != nil && cs != nil {
- // The first pass derives history once; later passes only what moved.
+ // The first pass derives history once. A later pass derives what
+ // either adapter moved, what waited on a lock, and what the removal
+ // found not derived yet. The last is how a file the deriver held for
+ // its grace is derived while a feed lands something on every pass.
var scope []string
if !r.full {
- scope = cs.Changed
+ scope = deriveScope(changed, r.derive)
}
if ms, derr := r.deriver.Pass(scope); derr != nil {
errs = append(errs, derr)
@@ -351,6 +485,30 @@ func (r *refresher) pass() error {
r.pushAfter = time.Now().Add(ps.RetryAfter)
}
}
+ // Remove what a scenario build marked and what is proved sent. It runs
+ // after the push, so a session can be sent and removed in one pass, and
+ // before the checks below, so the status the page records includes it.
+ removed := 0
+ var kept []remove.Kept
+ if r.remover != nil && r.scenarioLock != nil {
+ res := r.remover.Pass(r.removeInput())
+ errs = append(errs, res.Errors...)
+ kept = res.Kept
+ removed = len(res.Removed)
+ for _, id := range res.Removed {
+ // The page caches a folded conversation. A session carried in
+ // retry would be parsed again, and a parse of nothing is quiet,
+ // but there is nothing left to wait for.
+ if r.srv != nil {
+ r.srv.Forget(id)
+ }
+ delete(r.retry, id)
+ }
+ r.derive = map[string]bool{}
+ for _, id := range res.DeriveNext {
+ r.derive[id] = true
+ }
+ }
if r.last && len(r.retry) > 0 {
// No pass follows, so these are not deferred, they are unparsed.
errs = append(errs, fmt.Errorf("%d session(s) were held by another builder and are not parsed", len(r.retry)))
@@ -378,7 +536,10 @@ func (r *refresher) pass() error {
r.setStatus(st)
// A quiet pass every few seconds is not worth a line. The first one and
- // any that changed or failed are.
+ // any that changed, removed or failed are.
+ if removed > 0 {
+ pushed += fmt.Sprintf(" removed=%d", removed)
+ }
if first || landed > 0 || rounds > 0 || busy > 0 || pushed != "" || len(errs) > 0 {
contended := ""
if busy > 0 {
@@ -394,6 +555,17 @@ func (r *refresher) pass() error {
fmt.Fprintf(os.Stderr, " error: %v\n", e)
}
}
+ // A marked session kept for a reason a person has to act on. Each reason
+ // is said once per process, whether or not the pass line is printed. None
+ // is an error: a root that keeps a session it cannot prove sent is doing
+ // what it should.
+ for _, k := range kept {
+ who := k.Session
+ if who == "" {
+ who = "every marked session"
+ }
+ fmt.Fprintf(os.Stderr, " kept: %s: %s\n", who, k.Reason)
+ }
return errors.Join(errs...)
}
@@ -404,6 +576,23 @@ func (r *refresher) setStatus(st view.Status) {
}
}
+// deriveScope is what a later pass derives: the sessions that moved, and the
+// ones the removal carried. It is nil when both are empty, and a nil scope
+// derives every session, as a pass where nothing moved always has.
+func deriveScope(changed, carried map[string]bool) []string {
+ var out []string
+ for id := range changed {
+ out = append(out, id)
+ }
+ for id := range carried {
+ if !changed[id] {
+ out = append(out, id)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
+
// loop runs passes one interval apart, forever. A pass that fails is
// reported by the pass itself and the next one tries again.
func (r *refresher) loop() {
diff --git a/cmd/asz/scenario.go b/cmd/asz/scenario.go
index 8283f6f..33d86d8 100644
--- a/cmd/asz/scenario.go
+++ b/cmd/asz/scenario.go
@@ -31,7 +31,7 @@ import (
"github.com/apache/skywalking-ai-sessionizer/internal/scenario/run"
)
-const scenarioUsage = `asz scenario build FILE... --format {claude-code|sd} --out DIR [--at TIME] [--scale FACTOR] [--repeat N] [--every D] [--pick {cycle|random}] [--seed N] [--through CHECKPOINT]
+const scenarioUsage = `asz scenario build FILE... --format {claude-code|sd} --out DIR [--at TIME] [--scale FACTOR] [--repeat N] [--every D] [--pick {cycle|random}] [--seed N] [--through CHECKPOINT] [--remove {immediately|DURATION}]
asz scenario check FILE [--format {claude-code|sd|all}] [--out DIR] [--at TIME] [--scale FACTOR]
build writes a scenario's input into DIR and stops: for claude-code, the runtime's own files under
@@ -52,6 +52,11 @@ conversations keep arriving.
asz scenario build tests/scenarios --format claude-code --out DIR --every 30s --pick random
asz server -config DIR/asz.yaml
+A claude-code build also records, for each session, when the pipeline may remove it. asz collect
+and asz server remove a session once all of it has been sent to the configured receiver:
+immediately, or once its last record is older than --remove. A session no build wrote is never
+removed.
+
check builds, collects and parses at every checkpoint, compares the fold with FILE's expectation
file (NAME.expect.yaml), runs the properties every chain must have, and compares the formats with
each other. It exits non-zero on any failure and keeps its directory when one is given.
@@ -64,6 +69,8 @@ each other. It exits non-zero on any failure and keeps its directory when one is
--pick MODE cycle through the scenarios in order, or random (default cycle)
--seed N seeds --pick random; 0 varies with each run (default 0)
--through NAME build only the steps up to this checkpoint (build only)
+ --remove POLICY immediately (default), or a duration such as 24h or 7d counted from the
+ session's last record: when the pipeline removes a sent session (claude-code build only)
`
// cmdScenario handles both subcommands. It is dispatched before the common
@@ -85,6 +92,7 @@ func cmdScenario(args []string) int {
pick := fs.String("pick", "cycle", "cycle or random, when more than one scenario is given")
seed := fs.Int64("seed", 0, "seeds --pick random; 0 varies with each run")
through := fs.String("through", "", "build only through this checkpoint")
+ remove := fs.String("remove", scenario.PolicyImmediately, "immediately, or a duration such as 24h or 7d")
fs.Usage = func() { fmt.Fprint(os.Stderr, scenarioUsage) }
// A file may come before the flags, as the usage shows, or after them,
// and there may be several: parse, take the file the parse stopped on,
@@ -126,6 +134,19 @@ func cmdScenario(args []string) int {
fmt.Fprintf(os.Stderr, "asz: --pick %q; use %s or %s\n", *pick, pickCycle, pickRandom)
return 2
}
+ // Removal is a policy of the scenario, never an option of the
+ // product: each session's marker records it, and a pipeline reads it
+ // from there. An sd build writes no source and no marker.
+ if scenario.Format(*format) == scenario.FormatSD && given["remove"] {
+ fmt.Fprintln(os.Stderr, "asz: --remove applies to a claude-code build; an sd build writes no source, and its sessions are never removed")
+ return 2
+ }
+ removal, err := scenario.ParseRemove(*remove)
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "asz: --remove %q; use immediately, or a duration such as 30m, 24h or 7d\n", *remove)
+ return 2
+ }
+ marks := scenario.Format(*format) == scenario.FormatClaudeCode
set, err := scenario.LoadSet(files)
if err != nil {
fmt.Fprintln(os.Stderr, "asz:", err)
@@ -157,12 +178,15 @@ func cmdScenario(args []string) int {
// already written, and the collector, which reads forward from where
// it stopped, would land nothing for the second.
used := map[string]bool{}
- opts := scenario.Options{At: base, Scale: *scale, Interval: *interval, Through: *through, Watch: *every > 0}
+ opts := scenario.Options{At: base, Scale: *scale, Interval: *interval, Through: *through, Watch: *every > 0, Remove: removal}
if opts.At.IsZero() {
opts.At = time.Now()
}
if *every > 0 {
fmt.Printf("out : %s\nevery : %s\nscenarios: %d, %s\n", *out, *every, len(set), *pick)
+ if marks {
+ fmt.Printf("remove : %s\n", removal.Describe())
+ }
}
for k := 1; count == 0 || k <= count; k++ {
l := next(k)
@@ -221,10 +245,19 @@ func cmdScenario(args []string) int {
opts.At = last.Add(gap)
if k == count {
fmt.Printf("out : %s\nconfig : %s\n", b.Out, b.Config)
+ if marks {
+ fmt.Printf("remove : %s\n", removal.Describe())
+ }
}
}
return 0
case "check":
+ // A check builds and removes its own copies. A policy for a pipeline
+ // has nothing to act on here.
+ if given["remove"] {
+ fmt.Fprintln(os.Stderr, "asz: --remove is for scenario build")
+ return 2
+ }
var formats []scenario.Format
if *format != "" && *format != "all" {
formats = []scenario.Format{scenario.Format(*format)}
diff --git a/cmd/asz/scenario_test.go b/cmd/asz/scenario_test.go
index 5ab33e7..941d997 100644
--- a/cmd/asz/scenario_test.go
+++ b/cmd/asz/scenario_test.go
@@ -18,6 +18,8 @@
package main
import (
+ "os"
+ "path/filepath"
"testing"
"time"
@@ -137,3 +139,53 @@ func TestChooserRandomRepeatsWithASeed(t *testing.T) {
t.Fatal("random never left the first scenario")
}
}
+
+// TestTheRemovePolicyIsRefusedWhereItMeansNothing. --remove is a policy for
+// the pipeline over a claude-code build's root. An sd build writes no source
+// to remove, and a check keeps nothing, so both refuse it, as they refuse a
+// value that is no policy. Each is refused before anything is written.
+func TestTheRemovePolicyIsRefusedWhereItMeansNothing(t *testing.T) {
+ out := t.TempDir()
+ for name, args := range map[string][]string{
+ "an sd build": {"build", "x.yaml", "--format", "sd", "--out", out, "--remove", "1h"},
+ "a check": {"check", "x.yaml", "--remove", "1h"},
+ "a retention of nothing": {"build", "x.yaml", "--format", "claude-code", "--out", out, "--remove", "0"},
+ "a word that is no policy": {"build", "x.yaml", "--format", "claude-code", "--out", out, "--remove", "soon"},
+ } {
+ if code := cmdScenario(args); code != 2 {
+ t.Errorf("%s: exit status %d, want 2", name, code)
+ }
+ }
+ if items, err := os.ReadDir(out); err != nil || len(items) != 0 {
+ t.Fatalf("a refused build wrote %d entries: %v", len(items), err)
+ }
+}
+
+// TestABuildRecordsTheRemovePolicy. The policy a person gives reaches the
+// session's marker, which is the only place a pipeline reads it from.
+func TestABuildRecordsTheRemovePolicy(t *testing.T) {
+ out := t.TempDir()
+ file := filepath.Join("..", "..", "tests", "scenarios", "assembly.yaml")
+ if code := cmdScenario([]string{"build", file, "--format", "claude-code", "--out", out, "--at", "2026-01-01T00:00:00Z", "--remove", "7d"}); code != 0 {
+ t.Fatalf("exit status %d", code)
+ }
+ markers, err := filepath.Glob(filepath.Join(out, "_source", scenario.MarkerDir, "*.json"))
+ if err != nil || len(markers) != 1 {
+ t.Fatalf("%d markers: %v", len(markers), err)
+ }
+ m, err := scenario.ReadMarker(markers[0])
+ if err != nil {
+ t.Fatal(err)
+ }
+ if m.Policy != scenario.PolicyRetain || m.Retain != "168h0m0s" {
+ t.Fatalf("the marker says %q %q, want retain 168h0m0s", m.Policy, m.Retain)
+ }
+ // Without the flag, an sd build is not refused, and writes no marker.
+ sd := t.TempDir()
+ if code := cmdScenario([]string{"build", file, "--format", "sd", "--out", sd, "--at", "2026-01-01T00:00:00Z"}); code != 0 {
+ t.Fatalf("an sd build with no --remove exited %d", code)
+ }
+ if _, err := os.Stat(filepath.Join(sd, "_scenario")); !os.IsNotExist(err) {
+ t.Fatal("an sd build made a scenario root")
+ }
+}
diff --git a/cmd/asz/view.go b/cmd/asz/view.go
index 7203c04..58ca29a 100644
--- a/cmd/asz/view.go
+++ b/cmd/asz/view.go
@@ -22,6 +22,7 @@ import (
"net"
"net/http"
"os"
+ "path/filepath"
"strings"
"time"
@@ -218,12 +219,17 @@ func printPipeline(ref *refresher, cfg *config.Config) {
}
if ref.pusher == nil {
fmt.Println("export : none; set export.otlp.endpoint to send what is landed")
- return
+ } else {
+ o := cfg.Export.OTLP
+ endpoint := o.Endpoint + " over gRPC"
+ if o.Protocol == otlp.ProtocolHTTP {
+ endpoint = o.Endpoint + " over HTTP"
+ }
+ fmt.Printf("export : %s, at the end of every pass\n", endpoint)
}
- o := cfg.Export.OTLP
- endpoint := o.Endpoint + " over gRPC"
- if o.Protocol == otlp.ProtocolHTTP {
- endpoint = o.Endpoint + " over HTTP"
+ // A root a claude-code scenario build wrote. Only there does a pipeline
+ // remove anything, and only what a build marked.
+ if fi, err := os.Stat(filepath.Join(ref.zone.Root(), storage.ScenarioDir)); err == nil && fi.IsDir() {
+ fmt.Printf("scenario : %s\n", ref.removalSays())
}
- fmt.Printf("export : %s, at the end of every pass\n", endpoint)
}
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 1c864cd..b3e2330 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -8,6 +8,10 @@
on the record. It is what a token count is reported under, so a metric per model can be
produced from the landed data alone. Records landed before this carry none, and the glossary
says where the runtime writes it.
+- A parse takes the conversation's lock before it reads the session's index. A parser that read
+ the index first could hold an old one while a scenario removal took the session away, and then
+ publish a round over evidence that is gone. A parse of a session with no landed files and no
+ chain creates nothing.
## Commands
@@ -30,6 +34,21 @@
every `.yaml` in it, with `--pick cycle` or `--pick random` and `--seed` to repeat an order.
Each session is stamped so its last record lands when it was written, and carries an id no
earlier session has. It is a mock client to run beside `asz server` or `asz collect`.
+- `asz scenario build --remove` says when a pipeline may remove each session a `claude-code` build
+ writes: `immediately`, the default, or a duration such as `24h` or `7d` after the session's last
+ record. The build writes a marker for each session after every other file of it, naming the
+ policy and every file it wrote for the session with its size and SHA-256. `asz collect` and
+ `asz server` remove a marked session at the end of a pass, once every landed file, round and
+ metrics request of it is recorded as sent to the receiver they send to and none was rejected.
+ They remove its source files, its landed files, its chain, its spool files, its lines in the state
+ files, and last the marker. The session directory and its chain directory are each renamed into
+ `_removed/` in one call before they are deleted, so a crash never leaves half a session for a
+ parse to read. Only a pipeline whose storage
+ root is the build's `--out` removes. It holds `_scenario/.lock` there, and a second one waits.
+ When the configuration keeps every marked session, the `scenario :` line at the start says why.
+ Removal is a policy of the scenario, never of the product. The configuration has no removal
+ setting, and a session no build marked, which is every real Claude Code session, is never
+ removed.
## Read
@@ -200,6 +219,16 @@
files and rounds as logs and the metrics spool as metrics, each on its own, both on unless set
off. `asz push` says which it is sending, and a configuration with both off is refused.
+## Export
+
+- `push.state` names the receiver its files were sent to, on an `endpoint` line, and each file a
+ receiver rejected records of, on a `rejected` line. Both lines are kept on every save. A
+ `push.state` written before this names no receiver. It is read as sent to an unknown receiver, so
+ a scenario root that holds one removes nothing.
+- Over HTTP, a 2xx answer counts as sent only when its body is empty or protobuf, and a redirect is
+ not followed. Before, an HTML page answered with 200, or a redirect to a login page, left files
+ recorded as sent that no receiver stored.
+
## Documentation
- `design-notes/` is removed. What its three notes described that has landed is now documented in
diff --git a/docs/en/formats/storage-root.md b/docs/en/formats/storage-root.md
index 95e6c7d..f8bf8f7 100644
--- a/docs/en/formats/storage-root.md
+++ b/docs/en/formats/storage-root.md
@@ -42,15 +42,22 @@ session.
.lock
metrics-<received-at>-<seq>-otlp.pb one request the receiver adapter was sent
metrics-<session-id>-<seq>-local.pb the points derived from the session's landed file <seq>
- push.state what asz push has sent, with each file's digest
+ push.state what has been sent: each file's digest, the receivers,
+ and the files a receiver rejected records of
_export/
.lock one pusher per root
+ _scenario/ only in a root a claude-code scenario build wrote
+ .lock one pipeline per scenario root
+ _removed/ what a removal of a scenario session is deleting
+ <session-id>/ the session directory, moved here whole
+ <session-id>.chain/ its conversation's directory, moved here whole
```
Child streams are flat siblings of `main`, keyed by agent id. The storage path deliberately does
not mirror the source tree: a path must not encode a relationship the pipeline is supposed to
derive. [Export over OpenTelemetry](../setup/export-otlp.md) says how the metrics spool fills and
-what `push.state` records.
+what `push.state` records. `_scenario/` and `_removed/` exist only in a root a scenario build
+wrote. [Retention](#retention) says what they are for.
`liveness` in `session.state` is always `unknown`. asz has no check for whether a session is
still running.
@@ -140,7 +147,9 @@ state active
Every pass takes each append source through the same steps:
1. The file is gone. The cursor becomes `source_gone`, and the landed files stay. This is normal:
- Claude Code prunes its transcripts, and the landed files outlive them.
+ Claude Code prunes its transcripts, and the landed files outlive them. A session a scenario
+ build marked is the one exception. Once all of it is sent, a pipeline removes its source first
+ and then the whole session directory, as [Retention](#retention) describes.
2. The file is shorter than `offset`. It was cut short, and the source is a `conflict`.
3. The size equals `offset` and the identity is unchanged. Nothing is new, and nothing more is
read. A change is judged by the size, never by `mtime`, so clock skew and timestamp granularity
@@ -236,6 +245,12 @@ follow the head is refused, and a round file is created only when no file of tha
made read-only. A test in `tests/chain` runs four parsers at once on one session and requires
exactly one round.
+Parse takes the lock before it reads the index. A parser that read the index first could hold an
+old one while a scenario removal took the session away, and then publish a round over evidence
+that is gone. A parse of a session with no landed file and no round publishes nothing, and creates
+no chain directory when there was none. So a parse that runs after a removal leaves nothing
+behind.
+
## What travels
A storage root is complete on its own. A copy of it can be listed, verified, re-parsed and read on
@@ -280,3 +295,75 @@ landed sequence and row, so a round whose files are gone points at records nobod
[asz.view](asz-view.md) document is still served. Its `summary.state` is `incomplete`, and
`summary.problems` has one line for each missing file in each round. A node that stood on a missing
record has no text and no content state.
+
+### A scenario root
+
+asz removes a collected session on its own in one case only: a session a `claude-code` scenario
+build wrote. [Scenarios](../guides/scenario.md#removal-a-session-goes-once-it-is-sent) describes
+the policy. A real Claude Code session is never removed. Only the build writes the marker a removal
+needs, and the product configuration has no removal setting.
+
+The build writes one marker for each session, `DIR/_source/.asz-scenario/<session-id>.json`, and a
+pipeline reads it under the `source_root` of `claude-code-local`. The build writes it after every
+other file of the session, so the marker proves the build finished writing the session. It names
+the policy and lists every file the build wrote for the session, with its size and SHA-256. A
+removal deletes only the files listed there, and only while each still holds the bytes the build
+wrote. The build also creates `_scenario/` in the storage root, the sign that a pipeline over this
+root removes. Such a pipeline holds `_scenario/.lock` for as long as it runs, and a second one
+waits. It could otherwise land a removed session again through a source it listed a moment
+earlier, or parse evidence that is gone.
+
+A pipeline removes a marked session at the end of a pass, after the send, when all of these hold:
+
+- The adapters find exactly the files the marker lists, each still holding the bytes the build
+ wrote, and neither adapter would land anything more of them.
+- Every landed file of the session came from a file in the marker.
+- Every landed file, every round and every spool file of the session is in `push.state` with the
+ digest it has now, and none of them has a `rejected` line.
+- `push.state` names one receiver, the one this pipeline sends to.
+- The head round reaches the last landed file.
+- With `metrics` on, every landed file is derived.
+- The marker's policy allows it.
+- Nothing keeps every marked session. [Scenarios](../guides/scenario.md#removal-a-session-goes-once-it-is-sent)
+ lists what does, such as a source directory where Claude Code keeps its own files, or a source
+ directory discovery could not read.
+
+The removal then goes in this order:
+
+1. The marker is rewritten to say `removing`. From then on, a pass finishes the removal without
+ deciding again, and a build refuses to write the session.
+2. The source files the marker lists are deleted, then the empty directories named for the session.
+ A project directory, which every session of a feed shares, is never deleted.
+3. `<root>/<session-id>/` is renamed to `_removed/<session-id>/`, and deleted there.
+4. `_conversations/<session-id>/` is renamed to `_removed/<session-id>.chain/`, and deleted there.
+5. The spool files the session owns are deleted.
+6. The session's lines in `push.state` and `metrics.state` are dropped.
+7. The marker is deleted.
+
+Each step comes where it does for a reason:
+
+- **The source goes first.** The cursors live in `<root>/<session-id>/`. A source that outlived
+ them would be read again from byte 0, landed under new names and sent again, because
+ `push.state` is keyed by path.
+- **The landed files go before the chain.** With them gone, nothing parses the session again, and
+ every round left is already sent. The other way round, a restart would parse the landed files
+ into a new chain, and a round of it that differs from one already sent would be sent again.
+- **The state lines go last.** A line is dropped only once its file is gone, so no file is sent or
+ derived twice.
+- **The session directory and the chain directory each go in one rename.** Deleting a session
+ directory in place takes one call per
+ file, and a walk in name order reaches `index/` before `streams/`. A stop in the middle left a
+ session with no index and some of its landed files. Parse then rebuilt the index from what was
+ left and published a round of tombstones. A rename is one call. Before it the directory is whole,
+ and after it nothing that lists sessions or chains finds any of it: the session listings pass
+ over names that start with `_`, and `_removed/` is outside `_conversations/`. In a pipeline over
+ a scenario root, the removal step at the end of every pass first deletes whatever a stopped
+ removal left in `_removed/`. When `_removed` is a symbolic link, or not a directory, nothing is
+ deleted through it and nothing is moved into it. The pass then removes no session, and says why.
+
+Between the two renames the page still lists the conversation, because its rounds exist, and shows
+it as a conversation whose landed files are gone, as above. Once the chain is gone too, the page
+stops listing it, and the receiver holds the only copy. A test in `tests/chain` stops a removal at
+every point it reaches, including after each source file and each spool file it deletes, then runs
+the whole pipeline again as after a restart. Each time nothing is landed, derived, parsed or sent
+again, the session ends fully removed, and `_removed/` is empty.
diff --git a/docs/en/guides/scenario.md b/docs/en/guides/scenario.md
index 7cc2f76..b79442a 100644
--- a/docs/en/guides/scenario.md
+++ b/docs/en/guides/scenario.md
@@ -35,6 +35,10 @@ export:
endpoint: 127.0.0.1:11800
```
+Once a receiver is named, `asz collect` and `asz server` also remove each session a `claude-code`
+build wrote, once all of it is sent, as [Removal](#removal-a-session-goes-once-it-is-sent)
+describes. `asz push` sends and never removes.
+
The pushed records say where they came from. A `claude-code` build is landed by the Claude Code
adapter and is attributed to `Claude Code`; an `sd` build carries the `mock/1` dialect and is
attributed to `Mock Agent`, so a receiver never lists an invented conversation as a real one.
@@ -57,6 +61,7 @@ it, such as the export block, is kept across builds.
| `--pick MODE` | with a set of scenarios, `cycle` through them in order or take a `random` one |
| `--seed N` | seeds `--pick random`; `0`, the default, varies with each run |
| `--through NAME` | only the steps up to the checkpoint NAME |
+| `--remove POLICY` | `claude-code` only: when a pipeline removes a session once all of it is sent, `immediately` (default) or a duration such as `24h` or `7d` after its last record. See [Removal](#removal-a-session-goes-once-it-is-sent) |
With a fixed `--at`, every file is identical on every run.
@@ -73,7 +78,9 @@ asz server -config DIR/asz.yaml # collect, parse and
`server` refreshes on the interval because the configuration the build writes says `mode: watch`.
To feed a receiver instead of a page, name it under `export.otlp` and run `asz collect`, which
-sends at the end of every period.
+sends at the end of every period. With a receiver named, under either command, each session of the
+feed is removed once all of it is sent, or later with `--remove`.
+[Removal](#removal-a-session-goes-once-it-is-sent) says how.
Two things differ from a one-shot build, and both are what a live feed needs.
@@ -109,6 +116,74 @@ for a client anyway.
the last one. The period must be at least `1ms`: a feed id carries the moment it was written, in
milliseconds, so a shorter period could not give every session an id of its own.
+## Removal: a session goes once it is sent
+
+A `claude-code` build also says when a pipeline may remove each session it writes. It writes a
+marker for the session, `DIR/_source/.asz-scenario/<session-id>.json`, after every other file of
+it. The marker names the policy and lists each file the build wrote, with its size and SHA-256.
+`--remove` sets the policy:
+
+| `--remove` | A session goes at the first pass where all of it is sent and |
+| --- | --- |
+| `immediately`, the default | nothing more is needed |
+| a duration, such as `30m`, `24h` or `7d` | its last record is at least that old |
+
+In a watching pipeline, a session whose last call may go on in a later file waits two minutes
+before its metrics are derived, so it goes about two minutes after it lands. A single pass does not
+wait.
+
+`asz collect` and `asz server` remove a session at the end of a pass, after the send, once every
+landed file, every round and every metrics request of it is recorded as sent to the receiver they
+send to. They remove the source files the build wrote, the landed files, the chain, the spool files
+and the session's lines in the state files, and the marker last. Once removed, the conversation
+leaves the page, and the receiver holds the only copy.
+[Storage Root](../formats/storage-root.md#a-scenario-root) gives the order and the reason for each
+step.
+
+The pipeline removes only over the storage root the build wrote, `DIR` itself, where the build
+creates `_scenario/`. The `DIR/asz.yaml` the build writes sets `storage.root` to `DIR`. A pipeline
+with another `storage.root` lands and sends the sessions but never removes them, and prints no line
+about it.
+
+A duration counts from the session's last record, never from when it landed, so the same landed
+data always gives the same answer. A feed stamps each session so its last record falls when it was
+written, so a feed's session stays about that long after it was written. A one-shot build with an
+old `--at` is already past its duration, and goes as soon as it is sent. `d` counts whole days, as
+`metrics_lookback` does. One run applies one policy to every session it writes.
+
+The policy belongs to the scenario, never to the product. The configuration has no removal
+setting, and a session no build marked, which is every real Claude Code session, is never removed.
+An sd build writes no source and no marker, so its sessions are never removed, and `--remove` with
+`--format sd` is refused.
+
+Nothing is removed that the pipeline cannot prove sent. Each of these keeps every marked session,
+and the pass says why once, on standard error, as `kept: every marked session: <reason>`:
+
+- There is no `export.otlp.endpoint`, so nothing is sent.
+- `export.otlp.logs` or `export.otlp.metrics` is switched off.
+- The `claude-code-changes` adapter is switched off, or reads anything but
+ `DIR/_source/plugins/data`, where the build wrote the plugin's output.
+- `push.state` in the root records files sent to another receiver, or to one it cannot name.
+- The source directory is one of the places Claude Code keeps its files, holds one, or lies inside
+ one. All three are checked, whichever one the environment selects: what `CLAUDE_CONFIG_DIR`
+ names, `XDG_CONFIG_HOME/claude` and `~/.claude`.
+- `DIR/_removed` is a symbolic link, or not a directory.
+
+One session is kept, with a `kept: <session-id>: <reason>` line, when something about it needs a
+person. Examples are a receiver that rejected records of it, a file the build wrote that has
+changed or is missing, and a file for it that the build did not write. A kept line is not an error.
+Such a session stays until a person removes it. A session that is only not sent yet, or not old
+enough, is kept without a line.
+
+One pipeline works on a root at a time. A pipeline over a root a `claude-code` build wrote holds
+`DIR/_scenario/.lock` for as long as it runs, and a second one waits and says so.
+[collect](../setup/command-line.md#a-scenario-root) shows what each prints.
+
+A build refuses to write a session whose marker says a pipeline is removing it. Building a
+one-shot or `--repeat` scenario into the same directory again, after its sessions were removed,
+writes the same ids again. The pipeline lands and sends them as new, under new file names, so the
+receiver is sent them a second time. A feed never reuses an id.
+
## The scenario
```yaml
@@ -242,6 +317,7 @@ properties: # all on unless set false
cross_format: true
records_match: true
push_follows_the_wire: true
+ removed_after_sent: true
view_covers_the_session: true
parse:
max_round_bytes: 0 # a parse setting, when the scenario needs one
@@ -278,5 +354,15 @@ nothing, and writing every body back to its path must give a root that verifies
same. `push.kinds` names the file kinds a scenario's push must carry; `all-kinds.yaml` names all
six.
+The removal is checked too. `removed_after_sent` makes full copies of the finished root and runs a
+pipeline's pass over each: collect, derive, parse, send to a receiver in the test, then remove.
+Nothing is removed before a push, after a refused push, after a partial success, when `push.state`
+names another receiver, or when the marker is gone. After a clean push, `immediately` removes the
+whole session in one pass, and after that nothing is landed, derived, parsed or sent again. A
+retention of 24 hours keeps the session one second before its last record is 24 hours old, and
+removes it at that moment. An sd build is never marked. `removed-after-sent.yaml` holds every kind
+of file a removal deletes, and the tests in `tests/chain` stop its removal at every point it
+reaches.
+
The project's own tests are scenarios under `tests/scenarios/`, one property of assembly each,
run in both formats by `go test ./tests/`.
diff --git a/docs/en/setup/command-line.md b/docs/en/setup/command-line.md
index e6c44c7..d0bef33 100644
--- a/docs/en/setup/command-line.md
+++ b/docs/en/setup/command-line.md
@@ -76,18 +76,69 @@ One line per pass, and a quiet pass prints nothing:
| `busy` | how many times another process on the same root held a session, a conversation's chain or the export state this pass needed, absent when 0 |
| `pushed` | files and rounds sent, absent when no endpoint is named |
| `metrics` | spooled metrics requests sent |
+| `rejected` | records and points a receiver took and rejected, absent when 0. It also gives an `error:` line, because they are never sent again. |
+| `removed` | scenario sessions removed, absent when 0. See [A scenario root](#a-scenario-root). |
| `errors` | listed on standard error after the line |
`busy` is not an error. Running two pipelines on one root is supported, and the process that holds
-a lock is doing the same work. A session whose chain was held is parsed on a later pass. A single
-pass has no later pass, so there that session is listed as an error. So is an export state that
-another process still holds after two minutes of waiting, because then nothing was sent.
+a lock is doing the same work. A scenario root is the exception, below. A session whose chain was
+held is parsed on a later pass. A single pass has no later pass, so there that session is listed
+as an error. So is an export state that another process still holds after two minutes of waiting,
+because then nothing was sent.
Anything else the pass could not do is listed on standard error under the line, one `error:` line
each: a session that failed to parse, a request the receiver refused. A watching collector goes on
to the next pass. A single pass exits non-zero, so an unattended backfill can be read by its exit
status.
+### A scenario root
+
+A root a `claude-code` scenario build wrote holds `_scenario/`. There a pass does one more thing,
+after the send. It removes each session the build marked that is now sent in full, by the policy in
+the session's marker. [Removal](../guides/scenario.md#removal-a-session-goes-once-it-is-sent) says
+what that takes. A root no build wrote has no `_scenario/`, and nothing in it is ever removed.
+
+When the root holds `_scenario/` at the start, a `scenario :` line says what the removal will do.
+It asks the check each removal pass makes before it looks at any session, so it never promises a
+removal the configuration prevents. When nothing in the configuration prevents one, it prints:
+
+```text
+scenario : a session a build marked is removed by its policy once all of it is sent
+```
+
+Otherwise it prints `scenario : no session is removed: <reason>`, where the reason names what keeps
+every marked session, such as `no export endpoint; nothing is sent, so nothing is removed`. A root
+that gets `_scenario/` only later, from a feed started after the pipeline, has no such line at the
+start.
+
+Such a root takes one pipeline at a time. The first pass that finds `_scenario/` takes
+`_scenario/.lock` and holds it until the process ends. It prints the lock, then the same words as
+the line above:
+
+```text
+scenario : this pipeline holds <root>/_scenario/.lock; a session a build marked is removed by its policy once all of it is sent
+```
+
+A second watching pipeline over the same root prints `waiting : another pipeline holds
+<root>/_scenario/.lock; this one does nothing until it is released` once. It then lands, parses and
+sends nothing until the lock is free. A single pass waits up to two minutes for the lock, then exits
+non-zero with nothing done. A second pipeline that went on could land a removed session again, or
+parse evidence that is gone.
+
+A marked session the pass cannot prove sent is kept. When a person has to act, the reason is
+printed once per process on standard error:
+
+```text
+ kept: every marked session: no export endpoint; nothing is sent, so nothing is removed
+ kept: 3189c1f0-9ec4-4bd2-88dc-8eda88ac6db3: the receiver rejected records of 3189c1f0-…/streams/main/transcript-…-000001.sd
+```
+
+`every marked session` means the reason holds for all of them. A kept session is not an error, and
+a single pass does not exit non-zero for it. A session that is only not sent yet, or not yet old
+enough for its policy, is kept with no line. A removal that stops part way on a check, such as a
+source file that changed since the build wrote it, is an `error:` line that names the file. That
+session is left for a person.
+
## index
Reports the derived index of one session, or of every session: entries, joinable content blocks,
@@ -179,6 +230,9 @@ is enabled.
last pass, and the list page shows the same. The page is up before the first pass, so a large
backfill does not look like a hung command.
+Over a [scenario root](#a-scenario-root), it removes sent sessions as `collect` does, and the page
+stops listing each one it removes.
+
`server` needs a local source. To serve a root that already exists, use `view`.
## view
@@ -271,6 +325,7 @@ and of folding again while a conversation is still being written, has not been m
```text
asz scenario build FILE... --format {claude-code|sd} --out DIR [--at TIME] [--scale FACTOR] [--repeat N]
[--every D] [--pick {cycle|random}] [--seed N] [--through CHECKPOINT]
+ [--remove {immediately|DURATION}]
asz scenario check FILE [--format {claude-code|sd|all}] [--out DIR] [--at TIME] [--scale FACTOR]
```
@@ -292,16 +347,31 @@ session:
[14:22:31] assembly.yaml: c30736f2-ac0c-4a72-89b1-01a0844af62d (20 records, 15.4s)
```
+A `claude-code` build also writes a marker for each session, with the policy `--remove` names:
+`immediately`, the default, or a duration such as `30m`, `24h` or `7d`, counted from the session's
+last record. `asz collect` and `asz server` remove a marked session by that policy once all of it
+is sent, as [Removal](../guides/scenario.md#removal-a-session-goes-once-it-is-sent) describes. The
+build prints the policy after `config`, and a feed prints it in its header:
+
+```text
+remove : 24h after its last record, once sent
+```
+
+`--remove` with `--format sd` is refused, because an sd build writes no source and its sessions are
+never removed. `check` refuses `--remove`, and `build` refuses zero, a negative duration and
+anything it cannot read. Each refusal exits with status 2 and writes nothing.
+
## push
Sends every landed file and every round not yet sent to the OpenTelemetry logs receiver at
`export.otlp.endpoint`, one log record per file, then exits. One pass is all it does: a root that
keeps growing is sent by `asz collect`, which pushes at the end of every period. This command is
for a root that is already there, such as one copied from another machine or brought under a new
-budget by `asz repack`. One line for the pass:
+budget by `asz repack`. It never removes anything, even over a [scenario root](#a-scenario-root).
+One line for the pass:
```text
-[10:12:03] files=306 metrics=41 bytes=47.9MB wire=48.0MB requests=46 paused=0s errors=0 (1.1s)
+[10:12:03] files=306 metrics=41 bytes=47.9MB wire=48.0MB requests=46 paused=0s rejected=0 errors=0 (1.1s)
```
`metrics` counts the spooled metrics requests sent, `wire` is what went out, the requests as
diff --git a/docs/en/setup/export-otlp.md b/docs/en/setup/export-otlp.md
index 8e4fa15..581c9a5 100644
--- a/docs/en/setup/export-otlp.md
+++ b/docs/en/setup/export-otlp.md
@@ -34,6 +34,26 @@ Landed files and rounds are both write-once, so each is sent once. `push.state`
root lists what was sent, with the digest each file had; a file is recorded only after the request
carrying it succeeded, so a failed request leaves it for the next pass.
+`push.state` also names the receivers the files went to, with one `endpoint` line each:
+`grpc://host:port`, `grpcs://host:port` with `tls: true`, or the HTTP base URL. A `rejected` line
+names each file of a request a receiver took while it rejected some of its records, as
+[Transport](#transport) describes. Every save writes all three kinds, so neither new line is lost
+at the next request. An older asz passes over both, and drops them when it saves the file.
+
+```text
+schema 1
+updated_at 2026-09-11T10:00:00Z
+endpoint grpc://127.0.0.1:11800
+pushed <session-id>/streams/main/transcript-20260911T095959.000000000Z-000001.sd 9f2c…
+rejected <session-id>/streams/main/transcript-20260911T095959.000000000Z-000001.sd
+```
+
+Removing a [scenario session](../guides/scenario.md#removal-a-session-goes-once-it-is-sent)
+depends on these lines. A session goes only when `push.state` names exactly one receiver, the one
+the pipeline sends to, and no file of the session has a `rejected` line. A `push.state` that lists
+files and names no receiver was written before these lines existed, or saved by an older asz. It
+then names the receiver `unknown`, and no scenario session on that root is removed after that.
+
## Transport
The request is the `ExportLogsServiceRequest` of the OpenTelemetry protocol, built from the
@@ -49,7 +69,23 @@ logs receiver reads it.
`headers` travel with every request on both transports, as gRPC metadata or as HTTP headers,
which is where an authorization token goes. A receiver that answers with a partial success,
saying it rejected some records, has taken the request, and the protocol says not to send it
-again: the count is reported on the pass line as `rejected`, and the files are marked sent.
+again: the count is reported on the pass line as `rejected`, and the files are marked sent. Each
+file of that request also gets a `rejected` line in `push.state`, because the answer does not say
+which records were rejected. A scenario session with such a file is never removed.
+
+Over HTTP, a request counts as taken only when the answer has a 2xx status and a body that is empty
+or protobuf: `application/x-protobuf`, as OTLP names it, or `application/protobuf`, whatever the
+parameters. A 2xx answer with a body of any other type is an error. Nothing in the request is
+recorded as sent, so it goes again on the next pass:
+
+```text
+otlp: http://127.0.0.1:12800/v1/logs answered 200 OK with text/html, not an OTLP response; nothing is recorded as sent
+```
+
+A redirect is not followed. It is an error like any other answer outside 2xx, and a receiver that
+moved is configured again. Before this rule, an HTML page answered with 200, and a redirect to a
+login page, both left files recorded as sent that no receiver stored. The rule is the same for logs
+and metrics. Over gRPC, a call that returns without an error always carries the receiver's answer.
## What every record carries
@@ -200,7 +236,8 @@ and not the other is sent what it takes.
On the way out the resource is normalised to asz's identity, the service, the layer, the sender,
so the OAP holds one service for the runtime. A receiver that answers with a partial success has
taken the request, and the protocol says not to send it again: the rejected records or points are
-counted on the pass line as `rejected` and the file is marked sent.
+counted on the pass line as `rejected`, the file is marked sent, and `push.state` gets a `rejected`
+line for it.
## Rate
diff --git a/internal/adapters/claudecode/covered.go b/internal/adapters/claudecode/covered.go
new file mode 100644
index 0000000..4f93db0
--- /dev/null
+++ b/internal/adapters/claudecode/covered.go
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package claudecode
+
+import (
+ "crypto/sha256"
+ "encoding/hex"
+ "errors"
+ "fmt"
+ "os"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// Covered reports whether collecting s now would land nothing, because every
+// source of it is landed to its end. A scenario removal asks this before it
+// deletes a session's sources. A source deleted with a line not yet landed
+// would lose that line.
+//
+// Each source is read against the adapter's own cursor, as a pass reads it,
+// and nothing is written. The caller holds the session's lock, so no pass
+// moves a cursor meanwhile.
+//
+// ok is false with a reason, and no error, when a pass would land something
+// now: the session only has to wait. An error means a source cannot be
+// covered until a person looks at it. That is a cursor in conflict, a source
+// that conflicts with its cursor, a source that is gone, two sources that map
+// to one cursor, or a read that failed.
+func (c *Collector) Covered(s Session) (ok bool, reason string, err error) {
+ // Checked for every source before any is read. A pass refuses the second
+ // of two sources that map to one cursor, so that source is never landed,
+ // and the session must say so even while another source is still
+ // waiting to land.
+ claimed := make(map[string]string, len(s.Sources))
+ for _, src := range s.Sources {
+ dir, cursorPath, _ := c.cursorPaths(src)
+ if dir == "" {
+ return false, "", fmt.Errorf("claudecode: unknown source kind for %s", src.Rel)
+ }
+ if prev, dup := claimed[cursorPath]; dup && prev != src.Rel {
+ return false, "", fmt.Errorf("claudecode: %s and %s both map to %s, so one of them is never landed", prev, src.Rel, cursorPath)
+ }
+ claimed[cursorPath] = src.Rel
+ }
+ for _, src := range s.Sources {
+ _, cursorPath, _ := c.cursorPaths(src)
+ if src.Kind.Append() {
+ ok, reason, err = CoveredAppend(src.Path, src.Rel, cursorPath, c.MaxDelta)
+ } else {
+ ok, reason, err = coveredSnapshot(src, cursorPath)
+ }
+ if err != nil || !ok {
+ return ok, reason, err
+ }
+ }
+ return true, "", nil
+}
+
+// CoveredAppend reports whether an append source is landed to its end: a pass
+// would read no line from it now. Both adapters tail their sources with
+// TailAppend, so both apply this one rule. The results mean what they mean
+// for Covered.
+//
+// TailAppend decides, not a comparison of inodes. The same bytes under a new
+// identity are a move, not new data. A line not yet ended by a newline is
+// left behind by a pass, so a source that ends in one is not covered. A
+// 0-byte source with no cursor is covered, since there is nothing to land.
+func CoveredAppend(path, rel, cursorPath string, maxBytes int64) (bool, string, error) {
+ cur, err := storage.LoadCursor(cursorPath, storage.CursorAppend, rel)
+ if err != nil {
+ return false, "", err
+ }
+ if cur.State == storage.CursorConflict {
+ return false, "", fmt.Errorf("%s conflicts with its cursor, and collection stopped there", rel)
+ }
+ if maxBytes <= 0 {
+ maxBytes = 2 << 20
+ }
+ chunk, err := TailAppend(path, cur, maxBytes)
+ if err != nil {
+ if errors.Is(err, ErrSourceGone) {
+ return false, "", fmt.Errorf("%s is gone, so whether all of it landed cannot be told", rel)
+ }
+ return false, "", err
+ }
+ if len(chunk.Lines) > 0 || int64(chunk.NewOffset) != chunk.Size {
+ return false, rel + " is not landed to its end yet", nil
+ }
+ return true, "", nil
+}
+
+// coveredSnapshot reports whether a snapshot source would land nothing. That
+// is so when its body is empty, which lands nothing and saves no cursor, or
+// when its digest is the one its cursor recorded. It is the test landSnapshot
+// applies.
+func coveredSnapshot(src Source, cursorPath string) (bool, string, error) {
+ cur, err := storage.LoadCursor(cursorPath, storage.CursorSnapshot, src.Rel)
+ if err != nil {
+ return false, "", err
+ }
+ if cur.State == storage.CursorConflict {
+ return false, "", fmt.Errorf("%s conflicts with its cursor, and collection stopped there", src.Rel)
+ }
+ data, err := os.ReadFile(src.Path)
+ if err != nil {
+ if errors.Is(err, os.ErrNotExist) {
+ return false, "", fmt.Errorf("%s is gone, so whether it landed cannot be told", src.Rel)
+ }
+ return false, "", err
+ }
+ body := trimTrailingNewline(data)
+ if len(body) == 0 {
+ return true, "", nil
+ }
+ sum := sha256.Sum256(body)
+ if hex.EncodeToString(sum[:]) == cur.ContentSHA256 {
+ return true, "", nil
+ }
+ return false, src.Rel + " is not landed to its end yet", nil
+}
diff --git a/internal/adapters/claudecode/covered_test.go b/internal/adapters/claudecode/covered_test.go
new file mode 100644
index 0000000..712f10a
--- /dev/null
+++ b/internal/adapters/claudecode/covered_test.go
@@ -0,0 +1,210 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package claudecode_test
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// discoverOne is the one session a source root holds.
+func discoverOne(t *testing.T, src string) claudecode.Session {
+ t.Helper()
+ sessions, err := claudecode.Discover(src)
+ if err != nil || len(sessions) != 1 {
+ t.Fatalf("discovered %d sessions, want 1: %v", len(sessions), err)
+ }
+ return sessions[0]
+}
+
+func appendText(t *testing.T, path, text string) {
+ t.Helper()
+ f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0o644)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := f.WriteString(text); err != nil {
+ t.Fatal(err)
+ }
+ if err := f.Close(); err != nil {
+ t.Fatal(err)
+ }
+}
+
+// Covered holds exactly when a pass would land nothing: after a pass, and
+// not while a source has a line or a version that is not landed yet. A 0-byte
+// journal and an empty meta file never get a cursor, and are covered.
+func TestCoveredMeansAPassWouldLandNothing(t *testing.T) {
+ src, zone := t.TempDir(), t.TempDir()
+ base := filepath.Join(src, "-proj-a", tSess)
+ run := filepath.Join(base, "subagents", "workflows", "wf_r1")
+ main := filepath.Join(src, "-proj-a", tSess+".jsonl")
+ manifest := filepath.Join(base, "workflows", "wf_r1.json")
+ mk(t, main, "{\"uuid\":\"m1\"}\n")
+ mk(t, filepath.Join(base, "subagents", "agent-"+tAgent+".jsonl"), "{\"uuid\":\"c1\"}\n")
+ mk(t, filepath.Join(base, "subagents", "agent-"+tAgent+".meta.json"), `{"agentType":"general-purpose"}`)
+ mk(t, filepath.Join(run, "agent-a2222222222222222.jsonl"), "{\"uuid\":\"w1\"}\n")
+ mk(t, filepath.Join(run, "agent-a2222222222222222.meta.json"), "")
+ mk(t, filepath.Join(run, "journal.jsonl"), "")
+ mk(t, manifest, `{"runId":"wf_r1","status":"running"}`)
+ mk(t, filepath.Join(base, "workflows", "scripts", "my-flow-wf_r1.js"), "export const meta = {}\n")
+
+ col := claudecode.New(src, storage.NewZone(zone), 0)
+ covered := func() (bool, string) {
+ t.Helper()
+ ok, reason, err := col.Covered(discoverOne(t, src))
+ if err != nil {
+ t.Fatalf("Covered: %v", err)
+ }
+ return ok, reason
+ }
+ collect := func() {
+ t.Helper()
+ st, err := col.CollectAll(nil)
+ if err != nil || len(st.Errors) != 0 {
+ t.Fatalf("collect: %v %v", err, st.Errors)
+ }
+ }
+
+ if ok, reason := covered(); ok || !strings.Contains(reason, "is not landed to its end yet") {
+ t.Fatalf("before any pass: covered=%v reason=%q", ok, reason)
+ }
+ // Covered writes nothing, not even the session's directory.
+ if _, err := os.Stat(filepath.Join(zone, tSess)); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("Covered created the session's directory: %v", err)
+ }
+ collect()
+ if ok, reason := covered(); !ok {
+ t.Fatalf("after a pass: %s", reason)
+ }
+
+ // A line appended is not covered until a pass lands it.
+ appendText(t, main, "{\"uuid\":\"m2\"}\n")
+ if ok, reason := covered(); ok || !strings.Contains(reason, tSess+".jsonl") {
+ t.Fatalf("a line appended: covered=%v reason=%q", ok, reason)
+ }
+ collect()
+ if ok, reason := covered(); !ok {
+ t.Fatalf("after the line landed: %s", reason)
+ }
+
+ // A pass leaves a line with no newline behind, so that line is not landed.
+ appendText(t, main, "{\"uuid\":\"m3\"")
+ if ok, _ := covered(); ok {
+ t.Fatal("a line still being written reads as landed")
+ }
+ collect()
+ if ok, _ := covered(); ok {
+ t.Fatal("after a pass, a line still being written reads as landed")
+ }
+ appendText(t, main, "}\n")
+ collect()
+ if ok, reason := covered(); !ok {
+ t.Fatalf("after the finished line landed: %s", reason)
+ }
+
+ // A snapshot with new content is not covered until its version lands.
+ mk(t, manifest, `{"runId":"wf_r1","status":"completed"}`)
+ if ok, reason := covered(); ok || !strings.Contains(reason, "wf_r1.json") {
+ t.Fatalf("a rewritten manifest: covered=%v reason=%q", ok, reason)
+ }
+ collect()
+ if ok, reason := covered(); !ok {
+ t.Fatalf("after the new version landed: %s", reason)
+ }
+}
+
+// Where a pass would stop until a person looks, Covered says so with an
+// error and changes nothing. That is a source cut below its cursor, a source
+// that is gone, and two sources that map to one cursor.
+func TestCoveredIsAnErrorWhereAPassWouldStop(t *testing.T) {
+ src, zone := t.TempDir(), t.TempDir()
+ main := filepath.Join(src, "-proj-a", tSess+".jsonl")
+ mk(t, main, "{\"uuid\":\"a\"}\n{\"uuid\":\"b\"}\n")
+ col := claudecode.New(src, storage.NewZone(zone), 0)
+ if _, err := col.CollectAll(nil); err != nil {
+ t.Fatal(err)
+ }
+ cursor := filepath.Join(zone, tSess, "streams", "main", "transcript.cursor")
+ before, err := os.ReadFile(cursor)
+ if err != nil {
+ t.Fatal(err)
+ }
+
+ mk(t, main, "{\"uuid\":\"a\"}\n")
+ var ce *claudecode.ConflictError
+ if ok, _, err := col.Covered(discoverOne(t, src)); ok || !asConflict(err, &ce) {
+ t.Fatalf("a source cut below its cursor: covered=%v err=%v; want a conflict", ok, err)
+ }
+ if after, err := os.ReadFile(cursor); err != nil || string(after) != string(before) {
+ t.Fatalf("Covered changed the cursor: %v", err)
+ }
+
+ s := discoverOne(t, src)
+ if err := os.Remove(main); err != nil {
+ t.Fatal(err)
+ }
+ if ok, _, err := col.Covered(s); ok || err == nil || !strings.Contains(err.Error(), "is gone") {
+ t.Fatalf("a source that is gone: covered=%v err=%v", ok, err)
+ }
+
+ // Two child transcripts for one agent id, filed under two directories.
+ src2 := t.TempDir()
+ mk(t, filepath.Join(src2, "-proj-a", tSess, "subagents", "agent-"+tAgent+".jsonl"), "{\"uuid\":\"a\"}\n")
+ mk(t, filepath.Join(src2, "-proj-b", tSess, "subagents", "agent-"+tAgent+".jsonl"), "{\"uuid\":\"b\"}\n")
+ col2 := claudecode.New(src2, storage.NewZone(t.TempDir()), 0)
+ if ok, _, err := col2.Covered(discoverOne(t, src2)); ok || err == nil || !strings.Contains(err.Error(), "both map") {
+ t.Fatalf("two sources for one cursor: covered=%v err=%v", ok, err)
+ }
+}
+
+// The same bytes under a new inode are a move, not new data. A pass lands
+// nothing from them, so the session stays covered.
+func TestCoveredTakesAMoveForNoNewData(t *testing.T) {
+ src, zone := t.TempDir(), t.TempDir()
+ main := filepath.Join(src, "-proj", tSess+".jsonl")
+ mk(t, main, "{\"uuid\":\"a\"}\n{\"uuid\":\"b\"}\n")
+ col := claudecode.New(src, storage.NewZone(zone), 0)
+ if _, err := col.CollectAll(nil); err != nil {
+ t.Fatal(err)
+ }
+ cur, err := storage.LoadCursor(filepath.Join(zone, tSess, "streams", "main", "transcript.cursor"), storage.CursorAppend, "")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if cur.Ino == 0 {
+ t.Skip("no inode on this platform")
+ }
+ body, err := os.ReadFile(main)
+ if err != nil {
+ t.Fatal(err)
+ }
+ mk(t, main+".new", string(body))
+ if err := os.Rename(main+".new", main); err != nil {
+ t.Fatal(err)
+ }
+ if ok, reason, err := col.Covered(discoverOne(t, src)); !ok || err != nil {
+ t.Fatalf("the same bytes under a new inode: covered=%v reason=%q err=%v", ok, reason, err)
+ }
+}
diff --git a/internal/adapters/claudecode/tail.go b/internal/adapters/claudecode/tail.go
index 8afde27..b674fd6 100644
--- a/internal/adapters/claudecode/tail.go
+++ b/internal/adapters/claudecode/tail.go
@@ -32,7 +32,9 @@ import (
// ErrSourceGone means the source file no longer exists.
//
// This is an expected state, not a failure: Claude Code prunes transcripts, and
-// our landed deltas deliberately outlive them.
+// our landed deltas deliberately outlive them. For a real session they always
+// do. The one exception is a session a scenario build marked: once all of it
+// is sent, a pipeline removes its source and then its landed files.
var ErrSourceGone = errors.New("claudecode: source gone")
// ConflictError means the source changed underneath the cursor. Collection
diff --git a/internal/adapters/claudecodechanges/covered.go b/internal/adapters/claudecodechanges/covered.go
new file mode 100644
index 0000000..fadc547
--- /dev/null
+++ b/internal/adapters/claudecodechanges/covered.go
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package claudecodechanges
+
+import (
+ "fmt"
+ "path/filepath"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+)
+
+// Covered reports whether collecting s now would land nothing, because every
+// change file of it is landed to its end. The results mean what they mean for
+// the local adapter's Covered, and the rule is the same one, since both
+// adapters tail their files with claudecode.TailAppend. Nothing is written,
+// and the caller holds the session's lock.
+func (c *Collector) Covered(s Session) (bool, string, error) {
+ cursorOf := func(src Source) string {
+ return filepath.Join(c.Zone.StreamDir(src.Session, src.Stream), prefix+".cursor")
+ }
+ // Checked for every file before any is read. Two plugin directories can
+ // each hold a file for one stream. Both map to one cursor, and a pass
+ // would interleave them behind it, so neither can be said to be landed to
+ // its end, even while another file is still waiting to land.
+ claimed := make(map[string]string, len(s.Sources))
+ for _, src := range s.Sources {
+ cursorPath := cursorOf(src)
+ if prev, dup := claimed[cursorPath]; dup && prev != src.Rel {
+ return false, "", fmt.Errorf("claudecodechanges: %s and %s both map to %s, so neither can be said to be landed", prev, src.Rel, cursorPath)
+ }
+ claimed[cursorPath] = src.Rel
+ }
+ for _, src := range s.Sources {
+ ok, reason, err := claudecode.CoveredAppend(src.Path, src.Rel, cursorOf(src), c.MaxDelta)
+ if err != nil || !ok {
+ return ok, reason, err
+ }
+ }
+ return true, "", nil
+}
diff --git a/internal/adapters/claudecodechanges/covered_test.go b/internal/adapters/claudecodechanges/covered_test.go
new file mode 100644
index 0000000..202a58e
--- /dev/null
+++ b/internal/adapters/claudecodechanges/covered_test.go
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package claudecodechanges_test
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecodechanges"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// Covered holds exactly when a pass would land nothing from a session's
+// change files. A stream file with no record yet is 0 bytes, has nothing to
+// land, and is covered.
+func TestCoveredMeansAPassWouldLandNothing(t *testing.T) {
+ src := t.TempDir()
+ out := filepath.Join(src, "asz-changes-inline", "output", session)
+ main := filepath.Join(out, "main.jsonl")
+ appendTo(t, main, line("p1/c1", "main", "toolu_1", "/w"))
+ appendTo(t, filepath.Join(out, agent+".jsonl"), "")
+ zone := storage.NewZone(t.TempDir())
+ c := claudecodechanges.New(src, zone, 0)
+ covered := func() (bool, string) {
+ t.Helper()
+ sessions, err := claudecodechanges.Discover(src)
+ if err != nil || len(sessions) != 1 {
+ t.Fatalf("discovered %d sessions, want 1: %v", len(sessions), err)
+ }
+ ok, reason, err := c.Covered(sessions[0])
+ if err != nil {
+ t.Fatalf("Covered: %v", err)
+ }
+ return ok, reason
+ }
+ collect := func() {
+ t.Helper()
+ st, err := c.CollectAll(nil)
+ if err != nil || len(st.Errors) != 0 {
+ t.Fatalf("collect: %v %v", err, st.Errors)
+ }
+ }
+
+ if ok, reason := covered(); ok || !strings.Contains(reason, "main.jsonl is not landed to its end yet") {
+ t.Fatalf("before any pass: covered=%v reason=%q", ok, reason)
+ }
+ if _, err := os.Stat(zone.SessionDir(session)); !errors.Is(err, os.ErrNotExist) {
+ t.Fatalf("Covered created the session's directory: %v", err)
+ }
+ collect()
+ if ok, reason := covered(); !ok {
+ t.Fatalf("after a pass: %s", reason)
+ }
+
+ // A record written in two parts. A pass leaves the first part behind, so
+ // the session is covered only once a pass lands the whole line.
+ rec := line("p1/c2", "main", "toolu_2", "/w")
+ appendTo(t, main, rec[:20])
+ if ok, _ := covered(); ok {
+ t.Fatal("a line still being written reads as landed")
+ }
+ collect()
+ if ok, _ := covered(); ok {
+ t.Fatal("after a pass, a line still being written reads as landed")
+ }
+ appendTo(t, main, rec[20:])
+ if ok, _ := covered(); ok {
+ t.Fatal("a finished line no pass has taken reads as landed")
+ }
+ collect()
+ if ok, reason := covered(); !ok {
+ t.Fatalf("after the line landed: %s", reason)
+ }
+}
+
+// Two plugin directories can each hold a file for one stream. A pass would
+// interleave them behind one cursor, so neither is covered.
+func TestCoveredRefusesTwoFilesForOneStream(t *testing.T) {
+ src := t.TempDir()
+ appendTo(t, filepath.Join(src, "asz-changes-inline", "output", session, "main.jsonl"), line("p1/c1", "main", "toolu_1", "/w"))
+ appendTo(t, filepath.Join(src, "asz-changes-market", "output", session, "main.jsonl"), line("p2/c1", "main", "toolu_1", "/w"))
+ sessions, err := claudecodechanges.Discover(src)
+ if err != nil || len(sessions) != 1 || len(sessions[0].Sources) != 2 {
+ t.Fatalf("discover: %+v err=%v", sessions, err)
+ }
+ c := claudecodechanges.New(src, storage.NewZone(t.TempDir()), 0)
+ if ok, _, err := c.Covered(sessions[0]); ok || err == nil || !strings.Contains(err.Error(), "both map") {
+ t.Fatalf("two files for one stream: covered=%v err=%v", ok, err)
+ }
+}
diff --git a/internal/export/otlp/client.go b/internal/export/otlp/client.go
index 376947f..b385aaf 100644
--- a/internal/export/otlp/client.go
+++ b/internal/export/otlp/client.go
@@ -24,6 +24,7 @@ import (
"errors"
"fmt"
"io"
+ "mime"
"net/http"
"strconv"
"strings"
@@ -196,10 +197,14 @@ func newHTTP(o Options) (Client, error) {
if !strings.HasPrefix(o.Endpoint, "http://") && !strings.HasPrefix(o.Endpoint, "https://") {
return nil, fmt.Errorf("otlp: an http endpoint is a URL such as http://127.0.0.1:12800: %s", o.Endpoint)
}
+ // A redirect is not followed. A receiver that moved is configured again.
+ // A redirect to a login page, followed, answers 200 with a page, and that
+ // must never read as records taken.
+ noRedirect := func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }
return &httpClient{
base: strings.TrimRight(o.Endpoint, "/"),
headers: o.Headers,
- http: &http.Client{Timeout: o.timeout()},
+ http: &http.Client{Timeout: o.timeout(), CheckRedirect: noRedirect},
}, nil
}
@@ -231,10 +236,15 @@ func (c *httpClient) ExportMetrics(req *collmetricspb.ExportMetricsServiceReques
return out.GetPartialSuccess().GetRejectedDataPoints(), nil
}
-// post sends one request as protobuf and returns the answer's body when it
-// is an OTLP response, or nothing when the receiver answered with anything
-// else, such as an empty body from a proxy, which is a full success as far
-// as it says.
+// post sends one request as protobuf and returns the answer's body.
+//
+// A 2xx answer with an empty body is a full success, whatever its type,
+// since that is how a proxy in front of a receiver often answers. A 2xx
+// answer with a body is an OTLP response only when its type is protobuf.
+// Anything else is an error, and nothing is recorded as sent. Taking such an
+// answer as success was measured against a real listener: an HTML page
+// answered with 200, and a redirect to a login page, both left the files
+// marked sent while the receiver stored nothing.
func (c *httpClient) post(url string, req proto.Message) ([]byte, error) {
body, err := proto.Marshal(req)
if err != nil {
@@ -269,10 +279,24 @@ func (c *httpClient) post(url string, req proto.Message) ([]byte, error) {
}
return nil, fmt.Errorf("otlp: %s answered %s: %s", url, resp.Status, strings.TrimSpace(string(snippet)))
}
- if !strings.HasPrefix(resp.Header.Get("Content-Type"), "application/x-protobuf") {
+ if len(answer) == 0 {
return nil, nil
}
- return answer, nil
+ // OTLP names application/x-protobuf. A review of the OAP found that its
+ // HTTP handlers answer with application/protobuf, so both are taken, and
+ // any parameters are ignored.
+ contentType := resp.Header.Get("Content-Type")
+ media, _, _ := mime.ParseMediaType(contentType)
+ if media == "application/x-protobuf" || media == "application/protobuf" {
+ return answer, nil
+ }
+ if media == "" {
+ media = strings.TrimSpace(contentType)
+ }
+ if media == "" {
+ media = "no content type"
+ }
+ return nil, fmt.Errorf("otlp: %s answered %s with %s, not an OTLP response; nothing is recorded as sent", url, resp.Status, media)
}
func (c *httpClient) Close() error {
diff --git a/internal/export/otlp/client_test.go b/internal/export/otlp/client_test.go
new file mode 100644
index 0000000..db1398d
--- /dev/null
+++ b/internal/export/otlp/client_test.go
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package otlp_test
+
+import (
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ collogspb "go.opentelemetry.io/proto/otlp/collector/logs/v1"
+ collmetricspb "go.opentelemetry.io/proto/otlp/collector/metrics/v1"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
+)
+
+// httpReceiver serves one handler, and returns an HTTP client for it.
+func httpReceiver(t *testing.T, h http.HandlerFunc) otlp.Client {
+ t.Helper()
+ srv := httptest.NewServer(h)
+ t.Cleanup(srv.Close)
+ c, err := otlp.NewClient(otlp.Options{Protocol: otlp.ProtocolHTTP, Endpoint: srv.URL})
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = c.Close() })
+ return c
+}
+
+// An answer with no body is a full success whatever its type, since that is
+// how a proxy in front of a receiver often answers.
+func TestAnEmptyAnswerIsASuccessWhateverItsType(t *testing.T) {
+ c := httpReceiver(t, func(w http.ResponseWriter, _ *http.Request) {
+ w.Header().Set("Content-Type", "text/plain")
+ w.WriteHeader(http.StatusOK)
+ })
+ if n, err := c.Export(&collogspb.ExportLogsServiceRequest{}); err != nil || n != 0 {
+ t.Fatalf("logs: rejected %d, err %v; want a full success", n, err)
+ }
+ if n, err := c.ExportMetrics(&collmetricspb.ExportMetricsServiceRequest{}); err != nil || n != 0 {
+ t.Fatalf("metrics: rejected %d, err %v; want a full success", n, err)
+ }
+}
+
+// A protobuf answer is decoded under either name for the type, with any
+// parameters ignored, and its partial success is counted.
+func TestAProtobufAnswerGivesItsPartialSuccess(t *testing.T) {
+ for _, media := range []string{"application/x-protobuf", "application/protobuf", "application/protobuf; charset=binary"} {
+ c := httpReceiver(t, func(w http.ResponseWriter, r *http.Request) {
+ var out []byte
+ if r.URL.Path == "/v1/metrics" {
+ out, _ = proto.Marshal(&collmetricspb.ExportMetricsServiceResponse{
+ PartialSuccess: &collmetricspb.ExportMetricsPartialSuccess{RejectedDataPoints: 4}})
+ } else {
+ out, _ = proto.Marshal(&collogspb.ExportLogsServiceResponse{
+ PartialSuccess: &collogspb.ExportLogsPartialSuccess{RejectedLogRecords: 3}})
+ }
+ w.Header().Set("Content-Type", media)
+ _, _ = w.Write(out)
+ })
+ if n, err := c.Export(&collogspb.ExportLogsServiceRequest{}); err != nil || n != 3 {
+ t.Fatalf("%s: logs rejected %d, err %v; want 3", media, n, err)
+ }
+ if n, err := c.ExportMetrics(&collmetricspb.ExportMetricsServiceRequest{}); err != nil || n != 4 {
+ t.Fatalf("%s: metrics rejected %d, err %v; want 4", media, n, err)
+ }
+ }
+}
+
+// A page answered with 200, by a fallback route or a login page, is not an
+// OTLP response. It is an error, and a pass marks nothing sent, so the files
+// go again once the receiver answers properly.
+func TestAPageAnsweredWith200IsNotASuccess(t *testing.T) {
+ c := httpReceiver(t, func(w http.ResponseWriter, _ *http.Request) {
+ w.Header().Set("Content-Type", "text/html; charset=utf-8")
+ _, _ = w.Write([]byte("<html><body>Sign in</body></html>"))
+ })
+ _, err := c.Export(&collogspb.ExportLogsServiceRequest{})
+ if err == nil || !strings.Contains(err.Error(), "with text/html, not an OTLP response") {
+ t.Fatalf("an HTML page answered with 200 gave %v; want an error naming text/html", err)
+ }
+
+ z, _ := zoneWithOneSession(t)
+ p := &otlp.Pusher{Zone: z, Client: c, Version: "test", ServiceName: "Claude Code", Endpoint: "http://receiver"}
+ st, err := p.Pass()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if st.Files != 0 || len(st.Errors) == 0 {
+ t.Fatalf("files=%d errors=%v; want nothing marked and the error reported", st.Files, st.Errors)
+ }
+ sent, err := otlp.LoadSent(z.Root())
+ if err != nil {
+ t.Fatal(err)
+ }
+ for rel := range sentPaths(t, z) {
+ if _, ok := sent.Digest(rel); ok {
+ t.Fatalf("%s was recorded as sent", rel)
+ }
+ }
+}
+
+// A redirect is not followed. A receiver that moved is configured again, and
+// a redirect to a login page, followed, would answer 200 with a page.
+func TestARedirectIsNotFollowed(t *testing.T) {
+ var followed atomic.Int32
+ c := httpReceiver(t, func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path == "/login" {
+ followed.Add(1)
+ w.Header().Set("Content-Type", "text/html")
+ _, _ = w.Write([]byte("<html>login</html>"))
+ return
+ }
+ http.Redirect(w, r, "/login", http.StatusFound)
+ })
+ if _, err := c.Export(&collogspb.ExportLogsServiceRequest{}); err == nil || !strings.Contains(err.Error(), "302") {
+ t.Fatalf("a redirect gave %v; want an error naming the 302", err)
+ }
+ if n := followed.Load(); n != 0 {
+ t.Fatalf("the redirect's target was requested %d times", n)
+ }
+}
diff --git a/internal/export/otlp/otlptest/receiver.go b/internal/export/otlp/otlptest/receiver.go
index f6036e6..31fbd7d 100644
--- a/internal/export/otlp/otlptest/receiver.go
+++ b/internal/export/otlp/otlptest/receiver.go
@@ -17,7 +17,7 @@
// Package otlptest is an in-process OTLP logs receiver for tests. It serves
// both transports at once, keeps every request it accepted, and can be told
-// to refuse them. The scenario checks and the exporter's own tests use it,
+// to refuse them, or to take them and reject some of their records. The scenario checks and the exporter's own tests use it,
// so the same receiver reads what a pusher sent over either transport.
package otlptest
@@ -55,6 +55,9 @@ type Receiver struct {
// retryAfter over HTTP when it is set.
throttle bool
retryAfter time.Duration
+ // reject is how many records or data points every accepted request is
+ // answered as rejected, in a partial success.
+ reject int64
web *httptest.Server
rpc *grpc.Server
@@ -128,6 +131,16 @@ func (r *Receiver) Throttle(on bool, after time.Duration) {
r.throttle, r.retryAfter = on, after
}
+// Reject makes the receiver answer every request it accepts, over both
+// transports, with a partial success that rejects n records or data points.
+// Zero answers a full success again. The requests are kept either way, as a
+// receiver that took them keeps them.
+func (r *Receiver) Reject(n int64) {
+ r.mu.Lock()
+ defer r.mu.Unlock()
+ r.reject = n
+}
+
// errThrottled is the receiver asking to slow down.
var errThrottled = errors.New("slow down")
@@ -137,24 +150,46 @@ func (r *Receiver) Close() {
r.rpc.Stop()
}
-func (r *Receiver) accept(req *collogspb.ExportLogsServiceRequest) error {
+// accept keeps a request, or refuses it, and returns how many of its records
+// the answer rejects.
+func (r *Receiver) accept(req *collogspb.ExportLogsServiceRequest) (int64, error) {
r.mu.Lock()
defer r.mu.Unlock()
if err := r.refusal(); err != nil {
- return err
+ return 0, err
}
r.reqs = append(r.reqs, req)
- return nil
+ return r.reject, nil
}
-func (r *Receiver) acceptMetrics(req *collmetricspb.ExportMetricsServiceRequest) error {
+func (r *Receiver) acceptMetrics(req *collmetricspb.ExportMetricsServiceRequest) (int64, error) {
r.mu.Lock()
defer r.mu.Unlock()
if err := r.refusal(); err != nil {
- return err
+ return 0, err
}
r.metrics = append(r.metrics, req)
- return nil
+ return r.reject, nil
+}
+
+// logsAnswer is the answer to an accepted logs request: a full success, or a
+// partial one that rejects n records.
+func logsAnswer(n int64) *collogspb.ExportLogsServiceResponse {
+ if n <= 0 {
+ return &collogspb.ExportLogsServiceResponse{}
+ }
+ return &collogspb.ExportLogsServiceResponse{PartialSuccess: &collogspb.ExportLogsPartialSuccess{
+ RejectedLogRecords: n, ErrorMessage: "rejected by the test receiver"}}
+}
+
+// metricsAnswer is the answer to an accepted metrics request: a full
+// success, or a partial one that rejects n data points.
+func metricsAnswer(n int64) *collmetricspb.ExportMetricsServiceResponse {
+ if n <= 0 {
+ return &collmetricspb.ExportMetricsServiceResponse{}
+ }
+ return &collmetricspb.ExportMetricsServiceResponse{PartialSuccess: &collmetricspb.ExportMetricsPartialSuccess{
+ RejectedDataPoints: n, ErrorMessage: "rejected by the test receiver"}}
}
func (r *Receiver) refusal() error {
@@ -185,11 +220,12 @@ func (r *Receiver) serveHTTP(w http.ResponseWriter, req *http.Request) {
http.Error(w, "not an ExportMetricsServiceRequest: "+err.Error(), http.StatusBadRequest)
return
}
- if err := r.acceptMetrics(&msg); err != nil {
+ rejected, err := r.acceptMetrics(&msg)
+ if err != nil {
r.refuse(w, err)
return
}
- out, _ := proto.Marshal(&collmetricspb.ExportMetricsServiceResponse{})
+ out, _ := proto.Marshal(metricsAnswer(rejected))
w.Header().Set("Content-Type", "application/x-protobuf")
_, _ = w.Write(out)
return
@@ -199,11 +235,12 @@ func (r *Receiver) serveHTTP(w http.ResponseWriter, req *http.Request) {
http.Error(w, "not an ExportLogsServiceRequest: "+err.Error(), http.StatusBadRequest)
return
}
- if err := r.accept(&msg); err != nil {
+ rejected, err := r.accept(&msg)
+ if err != nil {
r.refuse(w, err)
return
}
- out, _ := proto.Marshal(&collogspb.ExportLogsServiceResponse{})
+ out, _ := proto.Marshal(logsAnswer(rejected))
w.Header().Set("Content-Type", "application/x-protobuf")
_, _ = w.Write(out)
}
@@ -230,13 +267,14 @@ type metricsServer struct {
}
func (s *metricsServer) Export(_ context.Context, req *collmetricspb.ExportMetricsServiceRequest) (*collmetricspb.ExportMetricsServiceResponse, error) {
- if err := s.r.acceptMetrics(req); err != nil {
+ rejected, err := s.r.acceptMetrics(req)
+ if err != nil {
if errors.Is(err, errThrottled) {
return nil, status.Error(codes.ResourceExhausted, err.Error())
}
return nil, status.Error(codes.Unavailable, err.Error())
}
- return &collmetricspb.ExportMetricsServiceResponse{}, nil
+ return metricsAnswer(rejected), nil
}
type logsServer struct {
@@ -245,13 +283,14 @@ type logsServer struct {
}
func (s *logsServer) Export(_ context.Context, req *collogspb.ExportLogsServiceRequest) (*collogspb.ExportLogsServiceResponse, error) {
- if err := s.r.accept(req); err != nil {
+ rejected, err := s.r.accept(req)
+ if err != nil {
if errors.Is(err, errThrottled) {
return nil, status.Error(codes.ResourceExhausted, err.Error())
}
return nil, status.Error(codes.Unavailable, err.Error())
}
- return &collogspb.ExportLogsServiceResponse{}, nil
+ return logsAnswer(rejected), nil
}
// Attrs flattens attributes for a check. An integer is written as int:N so
diff --git a/internal/export/otlp/push.go b/internal/export/otlp/push.go
index d62e440..3353328 100644
--- a/internal/export/otlp/push.go
+++ b/internal/export/otlp/push.go
@@ -63,6 +63,11 @@ type Pusher struct {
Zone *storage.Zone
// Client is the receiver, over either transport; see NewClient.
Client Client
+ // Endpoint names the receiver Client sends to, as EndpointOf gives it.
+ // push.state records it beside what was sent, so a reader can tell which
+ // receiver took the files. Empty records no receiver, and then nothing
+ // that asks where the files went trusts what this pusher sent.
+ Endpoint string
// Version is what the sender reports about itself.
Version string
// ServiceName is the service every record is attributed to, when one is
@@ -386,7 +391,12 @@ func (p *Pusher) pushSpool(b *batch) {
}
continue
}
- state.mark(rel, digestOf(data))
+ state.mark(rel, digestOf(data), p.Endpoint)
+ // A partial success is not sent again, as the protocol says. It is
+ // recorded, so no reader takes the file as received whole.
+ if rejected > 0 {
+ state.reject(rel)
+ }
st.Metrics++
if err := state.save(p.statePath(), p.Now()); err != nil {
st.Errors = append(st.Errors, err)
@@ -492,7 +502,7 @@ func (p *Pusher) sessionsOldestFirst() ([]session, error) {
return out, nil
}
-func (p *Pusher) statePath() string { return filepath.Join(p.Zone.Root(), "push.state") }
+func (p *Pusher) statePath() string { return filepath.Join(p.Zone.Root(), StateFile) }
// batch accumulates files and sends them when the budget is reached. A file
// is recorded as pushed only after the request carrying it succeeded, so a
@@ -585,7 +595,12 @@ func (b *batch) flush() error {
return err
}
for _, f := range pending {
- b.state.mark(f.rel, f.digest)
+ b.state.mark(f.rel, f.digest, b.p.Endpoint)
+ // The request was taken, so it is not sent again. The answer does not
+ // say which records were rejected, so every file in it is recorded.
+ if rejected > 0 {
+ b.state.reject(f.rel)
+ }
}
b.st.Files += len(pending)
return b.state.save(b.p.statePath(), b.p.Now())
@@ -1035,13 +1050,34 @@ func roundFiles(root, conv string) ([]string, error) {
return out, nil
}
-// pushState is the set of files already sent, with the digest each had.
+// StateFile is the file in a storage root that records what was sent.
+const StateFile = "push.state"
+
+// pushState is what push.state records: the files already sent, with the
+// digest each had, the receivers they went to, and the files of a request a
+// receiver took while it rejected some of the request's records.
+//
+// Every save writes all three. An older asz reads only the pushed lines, and
+// rewrites the file without the other two. So a file with pushed lines and
+// no endpoint line names the receiver unknown, and nothing that asks where
+// the files went trusts it.
type pushState struct {
- files map[string]string
+ files map[string]string
+ endpoints map[string]bool
+ rejected map[string]bool
+}
+
+// unknownEndpoint is the receiver of files that push.state records with no
+// endpoint: a file written before endpoints were recorded, or rewritten by an
+// older asz.
+const unknownEndpoint = "unknown"
+
+func newPushState() *pushState {
+ return &pushState{files: map[string]string{}, endpoints: map[string]bool{}, rejected: map[string]bool{}}
}
func loadState(path string) (*pushState, error) {
- s := &pushState{files: map[string]string{}}
+ s := newPushState()
f, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
@@ -1059,29 +1095,61 @@ func loadState(path string) (*pushState, error) {
if err != nil {
return nil, err
}
+ // The two newer line kinds have two fields. An older asz keeps only
+ // three-field pushed lines, so it passes over them.
fields := strings.Fields(string(line))
- if len(fields) == 3 && fields[0] == "pushed" {
+ switch {
+ case len(fields) == 3 && fields[0] == "pushed":
s.files[fields[1]] = fields[2]
+ case len(fields) == 2 && fields[0] == "endpoint":
+ s.endpoints[fields[1]] = true
+ case len(fields) == 2 && fields[0] == "rejected":
+ s.rejected[fields[1]] = true
}
}
+ if len(s.files) > 0 && len(s.endpoints) == 0 {
+ s.endpoints[unknownEndpoint] = true
+ }
return s, nil
}
-func (s *pushState) pushed(rel string) bool { _, ok := s.files[rel]; return ok }
-func (s *pushState) mark(rel, digest string) { s.files[rel] = digest }
+func (s *pushState) pushed(rel string) bool { _, ok := s.files[rel]; return ok }
-func (s *pushState) save(path string, now time.Time) error {
- keys := make([]string, 0, len(s.files))
- for k := range s.files {
- keys = append(keys, k)
+// mark records rel as sent with its digest, to endpoint when one is named.
+func (s *pushState) mark(rel, digest, endpoint string) {
+ s.files[rel] = digest
+ if endpoint != "" {
+ if s.endpoints == nil {
+ s.endpoints = map[string]bool{}
+ }
+ s.endpoints[endpoint] = true
}
- sort.Strings(keys)
+}
+
+// reject records that a receiver rejected records of rel.
+func (s *pushState) reject(rel string) {
+ if s.rejected == nil {
+ s.rejected = map[string]bool{}
+ }
+ s.rejected[rel] = true
+}
+
+// save writes every line kind, every time. flush and pushSpool save after
+// each request, so a kind left out here would be gone after the next request,
+// and a file a receiver rejected records of would then read as taken whole.
+func (s *pushState) save(path string, now time.Time) error {
return storage.WriteAtomic(path, storage.PermState, func(w io.Writer) error {
bw := bufio.NewWriter(w)
fmt.Fprintf(bw, "schema 1\nupdated_at %s\n", now.UTC().Format(time.RFC3339Nano))
- for _, k := range keys {
+ for _, e := range sortedKeys(s.endpoints) {
+ fmt.Fprintf(bw, "endpoint %s\n", e)
+ }
+ for _, k := range sortedKeys(s.files) {
fmt.Fprintf(bw, "pushed %s %s\n", k, s.files[k])
}
+ for _, k := range sortedKeys(s.rejected) {
+ fmt.Fprintf(bw, "rejected %s\n", k)
+ }
return bw.Flush()
})
}
diff --git a/internal/export/otlp/sent.go b/internal/export/otlp/sent.go
new file mode 100644
index 0000000..14e194c
--- /dev/null
+++ b/internal/export/otlp/sent.go
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package otlp
+
+import (
+ "errors"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "sort"
+ "strings"
+ "time"
+)
+
+// EndpointOf names a receiver the way push.state records it.
+//
+// A gRPC receiver is grpc://host:port, or grpcs://host:port over TLS. An
+// HTTP receiver is its base URL without trailing slashes, as the client keeps
+// it. A gRPC endpoint is never a URL, so the two forms never meet. An empty
+// endpoint names nothing.
+func EndpointOf(protocol, endpoint string, tls bool) string {
+ endpoint = strings.TrimSpace(endpoint)
+ if endpoint == "" {
+ return ""
+ }
+ switch protocol {
+ case ProtocolHTTP:
+ return strings.TrimRight(endpoint, "/")
+ case ProtocolGRPC, "":
+ if tls {
+ return "grpcs://" + endpoint
+ }
+ return "grpc://" + endpoint
+ }
+ return protocol + "://" + endpoint
+}
+
+// Sent is what push.state in a storage root records: the files sent, with
+// the digest each had, the files a receiver rejected records of, and the
+// receivers the files went to.
+type Sent struct{ state *pushState }
+
+// LoadSent reads push.state of a root. A root with none has sent nothing.
+func LoadSent(root string) (*Sent, error) {
+ s, err := loadState(filepath.Join(root, StateFile))
+ if err != nil {
+ return nil, err
+ }
+ return &Sent{state: s}, nil
+}
+
+// Digest is the digest rel had when it was sent, and whether it was sent.
+// rel is the file's path under the root, with forward slashes.
+func (s *Sent) Digest(rel string) (string, bool) {
+ d, ok := s.state.files[rel]
+ return d, ok
+}
+
+// Rejected reports whether a receiver took the request carrying rel and
+// said it rejected some of the request's records. The protocol says not to
+// send such a request again, so the file is recorded as sent all the same.
+func (s *Sent) Rejected(rel string) bool { return s.state.rejected[rel] }
+
+// Endpoints lists the receivers the recorded files were sent to, in order.
+// It names "unknown" when push.state records files and no receiver.
+func (s *Sent) Endpoints() []string { return sortedKeys(s.state.endpoints) }
+
+// SentTo reports whether rel was sent to endpoint with the digest given, and
+// no record of it was rejected.
+//
+// push.state records which receivers the files went to, not which file went
+// to which. So SentTo holds only when endpoint is the one receiver
+// push.state names. With two, which file reached which cannot be told, and
+// with the receiver unknown nothing can be told. An empty endpoint is never
+// sent to.
+func (s *Sent) SentTo(endpoint, rel, digest string) bool {
+ if endpoint == "" || endpoint == unknownEndpoint || len(s.state.endpoints) != 1 || !s.state.endpoints[endpoint] {
+ return false
+ }
+ d, ok := s.state.files[rel]
+ return ok && d == digest && !s.state.rejected[rel]
+}
+
+// ForgetGone drops the pushed and rejected lines of every path that owned
+// reports true for and whose file is gone. Every endpoint line stays. The
+// caller holds the root's export lock (storage.LockExport), because a pusher
+// rewrites the same file.
+//
+// A line is dropped only once its file is gone, and a pusher sends only files
+// that exist, so dropping a line sends nothing again. It would, if a file of
+// the same name appeared later, and ruling that out is the caller's job. A
+// root with no push.state is left without one.
+func ForgetGone(root string, owned func(rel string) bool, now time.Time) error {
+ if owned == nil {
+ return nil
+ }
+ path := filepath.Join(root, StateFile)
+ s, err := loadState(path)
+ if err != nil {
+ return err
+ }
+ changed := false
+ for rel := range s.files {
+ if owned(rel) && fileGone(root, rel) {
+ delete(s.files, rel)
+ changed = true
+ }
+ }
+ for rel := range s.rejected {
+ if owned(rel) && fileGone(root, rel) {
+ delete(s.rejected, rel)
+ changed = true
+ }
+ }
+ if !changed {
+ return nil
+ }
+ return s.save(path, now)
+}
+
+// fileGone reports whether the file at rel under root surely does not exist.
+// Any other answer, a permission error included, keeps its line.
+func fileGone(root, rel string) bool {
+ _, err := os.Lstat(filepath.Join(root, filepath.FromSlash(rel)))
+ return errors.Is(err, fs.ErrNotExist)
+}
+
+func sortedKeys[V any](m map[string]V) []string {
+ keys := make([]string, 0, len(m))
+ for k := range m {
+ keys = append(keys, k)
+ }
+ sort.Strings(keys)
+ return keys
+}
diff --git a/internal/export/otlp/sent_test.go b/internal/export/otlp/sent_test.go
new file mode 100644
index 0000000..a1acb00
--- /dev/null
+++ b/internal/export/otlp/sent_test.go
@@ -0,0 +1,309 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package otlp_test
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ collmetricspb "go.opentelemetry.io/proto/otlp/collector/metrics/v1"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// sentPaths is every file a push sends from zoneWithOneSession, by its path
+// under the root, with its digest.
+func sentPaths(t *testing.T, z *storage.Zone) map[string]string {
+ t.Helper()
+ files, err := storage.LandedFiles(z, "sess1")
+ if err != nil {
+ t.Fatal(err)
+ }
+ paths := []string{filepath.Join(z.Root(), "_conversations", "sess1", "rounds", "r000001-abcdefabcdef.sf")}
+ for _, lf := range files {
+ paths = append(paths, lf.Path)
+ }
+ out := map[string]string{}
+ for _, p := range paths {
+ rel, err := filepath.Rel(z.Root(), p)
+ if err != nil {
+ t.Fatal(err)
+ }
+ d, err := storage.FileDigest(p)
+ if err != nil {
+ t.Fatal(err)
+ }
+ out[filepath.ToSlash(rel)] = d
+ }
+ return out
+}
+
+// spool puts one metrics request in the root's spool, and returns its path
+// under the root and its digest.
+func spool(t *testing.T, z *storage.Zone) (string, string) {
+ t.Helper()
+ data, err := proto.Marshal(&collmetricspb.ExportMetricsServiceRequest{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ path, err := storage.NewSpool(z).Put("otlp", data, time.Now())
+ if err != nil {
+ t.Fatal(err)
+ }
+ rel, err := filepath.Rel(z.Root(), path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ d, err := storage.FileDigest(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return filepath.ToSlash(rel), d
+}
+
+// A receiver that takes a request and rejects some of its records has taken
+// it, and the protocol says not to send it again. So its files are recorded
+// as sent and as rejected. The rejected lines survive every later save, or
+// the files would read as received whole after the next request.
+func TestPushStateRecordsTheEndpointAndTheRejectedFiles(t *testing.T) {
+ for _, protocol := range protocols {
+ t.Run(protocol, func(t *testing.T) {
+ z, files := zoneWithOneSession(t)
+ rcv := startReceiver(t)
+ rcv.Reject(1)
+ endpoint := otlp.EndpointOf(protocol, rcv.Options(protocol).Endpoint, false)
+ p := &otlp.Pusher{Zone: z, Client: clientFor(t, rcv, protocol), Version: "test", ServiceName: "Claude Code", Endpoint: endpoint}
+ st, err := p.Pass()
+ if err != nil || len(st.Errors) != 0 || st.Files != len(files) || st.Rejected != 1 {
+ t.Fatalf("a partial success: files=%d rejected=%d errors=%v err=%v; want %d files marked and 1 rejected",
+ st.Files, st.Rejected, st.Errors, err, len(files))
+ }
+ first := sentPaths(t, z)
+ sent, err := otlp.LoadSent(z.Root())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := sent.Endpoints(); len(got) != 1 || got[0] != endpoint {
+ t.Fatalf("endpoints %v, want [%s]", got, endpoint)
+ }
+ for rel, digest := range first {
+ if d, ok := sent.Digest(rel); !ok || d != digest {
+ t.Fatalf("%s: recorded %q (%v), want its digest %s", rel, d, ok, digest)
+ }
+ if !sent.Rejected(rel) || sent.SentTo(endpoint, rel, digest) {
+ t.Fatalf("%s: a file of a request with rejected records must be recorded as rejected, never as sent whole", rel)
+ }
+ }
+ data, err := os.ReadFile(filepath.Join(z.Root(), otlp.StateFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(string(data), "\nendpoint "+endpoint+"\n") {
+ t.Fatalf("push.state carries no endpoint line for %s:\n%s", endpoint, data)
+ }
+
+ // A later request that rejects nothing saves push.state again.
+ rcv.Reject(0)
+ spoolRel, spoolDigest := spool(t, z)
+ st, err = p.Pass()
+ if err != nil || len(st.Errors) != 0 || st.Metrics != 1 {
+ t.Fatalf("the second pass: metrics=%d errors=%v err=%v", st.Metrics, st.Errors, err)
+ }
+ if sent, err = otlp.LoadSent(z.Root()); err != nil {
+ t.Fatal(err)
+ }
+ for rel := range first {
+ if !sent.Rejected(rel) {
+ t.Fatalf("%s: the rejected line was lost at the next save", rel)
+ }
+ }
+ if sent.Rejected(spoolRel) || !sent.SentTo(endpoint, spoolRel, spoolDigest) {
+ t.Fatalf("%s was taken whole, and must read as sent to %s", spoolRel, endpoint)
+ }
+ if sent.SentTo(endpoint, spoolRel, strings.Repeat("0", 64)) {
+ t.Fatal("a different digest read as sent")
+ }
+ })
+ }
+}
+
+// push.state records which receivers the files went to, not which file went
+// to which. A root sent to two receivers proves nothing about either.
+func TestSentToNeedsTheOneReceiverPushStateNames(t *testing.T) {
+ z, files := zoneWithOneSession(t)
+ rcv := startReceiver(t)
+ a := otlp.EndpointOf(otlp.ProtocolGRPC, rcv.Options(otlp.ProtocolGRPC).Endpoint, false)
+ p := &otlp.Pusher{Zone: z, Client: clientFor(t, rcv, otlp.ProtocolGRPC), Version: "test", ServiceName: "Claude Code", Endpoint: a}
+ if st, err := p.Pass(); err != nil || st.Files != len(files) {
+ t.Fatalf("files=%d err=%v", st.Files, err)
+ }
+ sent, err := otlp.LoadSent(z.Root())
+ if err != nil {
+ t.Fatal(err)
+ }
+ for rel, digest := range sentPaths(t, z) {
+ if !sent.SentTo(a, rel, digest) {
+ t.Fatalf("%s must read as sent to %s", rel, a)
+ }
+ if sent.SentTo("grpc://elsewhere:11800", rel, digest) || sent.SentTo("", rel, digest) {
+ t.Fatalf("%s reads as sent to a receiver it never reached", rel)
+ }
+ }
+
+ // The same root, sent on to a second receiver.
+ b := otlp.EndpointOf(otlp.ProtocolHTTP, rcv.Options(otlp.ProtocolHTTP).Endpoint, false)
+ spoolRel, spoolDigest := spool(t, z)
+ p = &otlp.Pusher{Zone: z, Client: clientFor(t, rcv, otlp.ProtocolHTTP), Version: "test", ServiceName: "Claude Code", Endpoint: b}
+ if st, err := p.Pass(); err != nil || st.Metrics != 1 {
+ t.Fatalf("metrics=%d err=%v", st.Metrics, err)
+ }
+ if sent, err = otlp.LoadSent(z.Root()); err != nil {
+ t.Fatal(err)
+ }
+ if got := sent.Endpoints(); len(got) != 2 {
+ t.Fatalf("endpoints %v, want both receivers", got)
+ }
+ if sent.SentTo(b, spoolRel, spoolDigest) || sent.SentTo(a, spoolRel, spoolDigest) {
+ t.Fatal("with two receivers recorded, which file reached which cannot be told")
+ }
+}
+
+// A push.state in today's format, with pushed lines only, still loads. The
+// receiver it names is unknown, so it proves nothing about any receiver. A
+// pusher with no endpoint leaves the same.
+func TestPushStateWithNoEndpointNamesTheReceiverUnknown(t *testing.T) {
+ root := t.TempDir()
+ today := "schema 1\nupdated_at 2026-09-10T00:00:00Z\n" +
+ "pushed a/streams/main/x.sd 1111\npushed _conversations/a/rounds/r000001-aaaaaaaaaaaa.sf 2222\n"
+ if err := os.WriteFile(filepath.Join(root, otlp.StateFile), []byte(today), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ sent, err := otlp.LoadSent(root)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if d, ok := sent.Digest("a/streams/main/x.sd"); !ok || d != "1111" {
+ t.Fatalf("a push.state in today's format did not load: %q %v", d, ok)
+ }
+ if got := sent.Endpoints(); len(got) != 1 || got[0] != "unknown" {
+ t.Fatalf("endpoints %v, want [unknown]", got)
+ }
+ if sent.SentTo("unknown", "a/streams/main/x.sd", "1111") || sent.SentTo("grpc://oap:11800", "a/streams/main/x.sd", "1111") {
+ t.Fatal("a file sent to an unknown receiver reads as sent to one")
+ }
+
+ z, _ := zoneWithOneSession(t)
+ rcv := startReceiver(t)
+ p := &otlp.Pusher{Zone: z, Client: clientFor(t, rcv, otlp.ProtocolGRPC), Version: "test", ServiceName: "Claude Code"}
+ if _, err := p.Pass(); err != nil {
+ t.Fatal(err)
+ }
+ data, err := os.ReadFile(filepath.Join(z.Root(), otlp.StateFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ if strings.Contains(string(data), "\nendpoint ") {
+ t.Fatalf("a pusher with no endpoint wrote one:\n%s", data)
+ }
+ if sent, err = otlp.LoadSent(z.Root()); err != nil {
+ t.Fatal(err)
+ }
+ if got := sent.Endpoints(); len(got) != 1 || got[0] != "unknown" {
+ t.Fatalf("endpoints %v, want [unknown]", got)
+ }
+
+ empty, err := otlp.LoadSent(t.TempDir())
+ if err != nil || len(empty.Endpoints()) != 0 {
+ t.Fatalf("a root that sent nothing: endpoints %v, err %v", empty.Endpoints(), err)
+ }
+ if _, ok := empty.Digest("a/streams/main/x.sd"); ok {
+ t.Fatal("a root that sent nothing reads as having sent a file")
+ }
+}
+
+// ForgetGone drops only the lines of the files it is told are owned and that
+// are gone. Every other line, and every endpoint line, stays.
+func TestForgetGoneDropsOnlyOwnedFilesThatAreGone(t *testing.T) {
+ root := t.TempDir()
+ if err := os.MkdirAll(filepath.Join(root, "a"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(filepath.Join(root, "a", "kept.sd"), []byte("x"), 0o444); err != nil {
+ t.Fatal(err)
+ }
+ state := "schema 1\nupdated_at 2026-09-10T00:00:00Z\nendpoint grpc://oap:11800\n" +
+ "pushed a/kept.sd d1\npushed a/gone.sd d2\npushed b/gone.sd d3\n" +
+ "rejected a/gone.sd\nrejected b/gone.sd\n"
+ if err := os.WriteFile(filepath.Join(root, otlp.StateFile), []byte(state), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ owned := func(rel string) bool { return strings.HasPrefix(rel, "a/") }
+ if err := otlp.ForgetGone(root, owned, time.Now()); err != nil {
+ t.Fatal(err)
+ }
+ sent, err := otlp.LoadSent(root)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, ok := sent.Digest("a/kept.sd"); !ok {
+ t.Fatal("the line of a file that is still there was dropped")
+ }
+ if _, ok := sent.Digest("a/gone.sd"); ok || sent.Rejected("a/gone.sd") {
+ t.Fatal("the lines of an owned file that is gone stayed")
+ }
+ if _, ok := sent.Digest("b/gone.sd"); !ok || !sent.Rejected("b/gone.sd") {
+ t.Fatal("the lines of a file nobody owns were dropped")
+ }
+ if got := sent.Endpoints(); len(got) != 1 || got[0] != "grpc://oap:11800" {
+ t.Fatalf("endpoints %v, want the one recorded", got)
+ }
+
+ bare := t.TempDir()
+ if err := otlp.ForgetGone(bare, owned, time.Now()); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := os.Stat(filepath.Join(bare, otlp.StateFile)); !os.IsNotExist(err) {
+ t.Fatalf("ForgetGone made a push.state in a root that had none: %v", err)
+ }
+}
+
+// A receiver is named by its transport and address, so the same receiver
+// reached the same way always has the same name.
+func TestEndpointOfNamesTheReceiver(t *testing.T) {
+ for _, c := range []struct {
+ protocol, endpoint string
+ tls bool
+ want string
+ }{
+ {otlp.ProtocolGRPC, "oap:11800", false, "grpc://oap:11800"},
+ {"", "oap:11800", false, "grpc://oap:11800"},
+ {otlp.ProtocolGRPC, "oap:11800", true, "grpcs://oap:11800"},
+ {otlp.ProtocolHTTP, "http://oap:12800/", false, "http://oap:12800"},
+ {otlp.ProtocolHTTP, "https://oap:12800//", true, "https://oap:12800"},
+ {otlp.ProtocolGRPC, "", false, ""},
+ } {
+ if got := otlp.EndpointOf(c.protocol, c.endpoint, c.tls); got != c.want {
+ t.Errorf("EndpointOf(%q, %q, %v) = %q, want %q", c.protocol, c.endpoint, c.tls, got, c.want)
+ }
+ }
+}
diff --git a/internal/metrics/metrics.go b/internal/metrics/metrics.go
index cb1783e..2273a28 100644
--- a/internal/metrics/metrics.go
+++ b/internal/metrics/metrics.go
@@ -229,8 +229,7 @@ func (d *Deriver) Pass(sessions []string) (*Stats, error) {
st.Errors = append(st.Errors, err)
continue
}
- name := fmt.Sprintf("metrics-%s-%06d-%s.pb", session, lf.Seq, SourceLocal)
- _, created, err := spool.PutNamed(name, data)
+ _, created, err := spool.PutNamed(SpoolName(session, lf.Seq), data)
if err != nil {
st.Errors = append(st.Errors, err)
continue
diff --git a/internal/metrics/session.go b/internal/metrics/session.go
new file mode 100644
index 0000000..6a09304
--- /dev/null
+++ b/internal/metrics/session.go
@@ -0,0 +1,172 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package metrics
+
+import (
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// SpoolName is the name of the spool file the deriver writes for one landed
+// file: the session, then the landed file's sequence, then the source.
+// SpoolOwner reads it back.
+func SpoolName(session string, seq uint64) string {
+ return fmt.Sprintf("metrics-%s-%06d-%s.pb", session, seq, SourceLocal)
+}
+
+// SpoolOwner returns the session a spool file the deriver wrote belongs to.
+//
+// The name is read from the right. The sequence is the last dash-separated
+// part before the source, and it has at least six digits. Everything before
+// it is the session. A pattern such as metrics-<session>-* is never used,
+// because one session id can be another followed by a dash and digits, and
+// the other's files would then be taken for its own. A request the receiver
+// adapter put in the spool names no session, so no session owns it.
+func SpoolOwner(name string) (string, bool) {
+ rest, ok := strings.CutPrefix(name, "metrics-")
+ if !ok {
+ return "", false
+ }
+ rest, ok = strings.CutSuffix(rest, "-"+SourceLocal+".pb")
+ if !ok {
+ return "", false
+ }
+ i := strings.LastIndexByte(rest, '-')
+ if i <= 0 {
+ return "", false
+ }
+ seq := rest[i+1:]
+ if len(seq) < 6 {
+ return "", false
+ }
+ for _, c := range seq {
+ if c < '0' || c > '9' {
+ return "", false
+ }
+ }
+ return rest[:i], true
+}
+
+// Derived is what metrics.state records as derived: every landed file the
+// deriver has read, by its path under the root, with the SHA-256 it had.
+type Derived struct{ files map[string]string }
+
+// LoadDerived reads metrics.state of a root once. A root with none has
+// derived nothing.
+func LoadDerived(z *storage.Zone) (*Derived, error) {
+ s, _, err := loadState(filepath.Join(storage.NewSpool(z).Dir(), StateFile))
+ if err != nil {
+ return nil, err
+ }
+ return &Derived{files: s.Derived}, nil
+}
+
+// Has reports whether the landed file at rel, its path under the root with
+// forward slashes, was derived with the digest given.
+func (d *Derived) Has(rel, digest string) bool {
+ got, ok := d.files[rel]
+ return ok && got == digest
+}
+
+// DerivedAll reports whether every landed file of a session was derived with
+// the digest it has now. files may be nil, and then the session's landed
+// files are listed. It reads metrics.state on every call, so a caller that
+// asks about many sessions calls LoadDerived once and uses Has.
+func DerivedAll(z *storage.Zone, session string, files []storage.LandedFile) (bool, error) {
+ if files == nil {
+ var err error
+ if files, err = storage.LandedFiles(z, session); err != nil {
+ return false, err
+ }
+ }
+ d, err := LoadDerived(z)
+ if err != nil {
+ return false, err
+ }
+ for _, lf := range files {
+ rel, err := filepath.Rel(z.Root(), lf.Path)
+ if err != nil {
+ return false, err
+ }
+ digest, err := storage.FileDigest(lf.Path)
+ if err != nil {
+ return false, err
+ }
+ if !d.Has(filepath.ToSlash(rel), digest) {
+ return false, nil
+ }
+ }
+ return true, nil
+}
+
+// Forget drops what metrics.state says about sessions that were removed:
+// every derived entry under <session>/ whose landed file is gone, and the
+// session's counted calls and series once no derived entry of it is left.
+//
+// An entry is dropped only when its file is gone, so no file is derived twice
+// for it. A session with a file left keeps its counted calls, so a call cut
+// across two files is never counted twice. The deriver makes a session's
+// state only for a session it lists, and it never lists a session whose
+// directory is gone, so the state is not made again. The caller makes sure
+// no deriver runs over the root at the same time. A root with no
+// metrics.state is left without one.
+func Forget(z *storage.Zone, sessions []string, now time.Time) error {
+ path := filepath.Join(storage.NewSpool(z).Dir(), StateFile)
+ s, first, err := loadState(path)
+ if err != nil {
+ return err
+ }
+ if first {
+ return nil
+ }
+ changed := false
+ for _, id := range sessions {
+ if id == "" {
+ continue
+ }
+ prefix := id + "/"
+ left := false
+ for rel := range s.Derived {
+ if !strings.HasPrefix(rel, prefix) {
+ continue
+ }
+ _, err := os.Lstat(filepath.Join(z.Root(), filepath.FromSlash(rel)))
+ if errors.Is(err, fs.ErrNotExist) {
+ delete(s.Derived, rel)
+ changed = true
+ continue
+ }
+ left = true
+ }
+ if _, ok := s.Sessions[id]; ok && !left {
+ delete(s.Sessions, id)
+ changed = true
+ }
+ }
+ if !changed {
+ return nil
+ }
+ return s.save(path, now)
+}
diff --git a/internal/metrics/session_test.go b/internal/metrics/session_test.go
new file mode 100644
index 0000000..a426baa
--- /dev/null
+++ b/internal/metrics/session_test.go
@@ -0,0 +1,223 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package metrics_test
+
+import (
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/metrics"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// readMetricsState reads metrics.state as a test sees it.
+func readMetricsState(t *testing.T, z *storage.Zone) (derived map[string]string, sessions map[string]json.RawMessage) {
+ t.Helper()
+ data, err := os.ReadFile(filepath.Join(storage.NewSpool(z).Dir(), metrics.StateFile))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var s struct {
+ Derived map[string]string `json:"derived"`
+ Sessions map[string]json.RawMessage `json:"sessions"`
+ }
+ if err := json.Unmarshal(data, &s); err != nil {
+ t.Fatal(err)
+ }
+ return s.Derived, s.Sessions
+}
+
+// relUnder is a landed file's path under the root, as metrics.state keys it.
+func relUnder(t *testing.T, z *storage.Zone, path string) string {
+ t.Helper()
+ rel, err := filepath.Rel(z.Root(), path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return filepath.ToSlash(rel)
+}
+
+// SpoolOwner reads a name from the right, so no session owns another's
+// files. The ids abc and abc-1 are the case a pattern gets wrong.
+func TestSpoolOwnerReadsTheNameFromTheRight(t *testing.T) {
+ for _, c := range []struct {
+ name, owner string
+ ok bool
+ }{
+ {metrics.SpoolName("abc", 1), "abc", true},
+ {metrics.SpoolName("abc-1", 2), "abc-1", true},
+ {"metrics-abc-1-000002-local.pb", "abc-1", true},
+ {metrics.SpoolName("c30736f2-ac0c-4a72-89b1-01a0844af62d", 1234567), "c30736f2-ac0c-4a72-89b1-01a0844af62d", true},
+ // A request the receiver adapter received names no session.
+ {"metrics-20260911T100000.000000000Z-000003-otlp.pb", "", false},
+ // The deriver writes at least six digits.
+ {"metrics-abc-00001-local.pb", "", false},
+ {"metrics-abc-00000x-local.pb", "", false},
+ {"metrics--000001-local.pb", "", false},
+ {"metrics-000001-local.pb", "", false},
+ {"spool.state", "", false},
+ {metrics.StateFile, "", false},
+ } {
+ owner, ok := metrics.SpoolOwner(c.name)
+ if owner != c.owner || ok != c.ok {
+ t.Errorf("SpoolOwner(%q) = %q, %v; want %q, %v", c.name, owner, ok, c.owner, c.ok)
+ }
+ }
+}
+
+// The deriver names its files the way SpoolOwner reads them back.
+func TestTheDeriverNamesItsFilesForTheirOwner(t *testing.T) {
+ z := storage.NewZone(t.TempDir())
+ land(t, z, "abc", "main", 1, []call{{id: "c1", model: "m", at: base, frags: 1, in: 1, out: 1}})
+ land(t, z, "abc-1", "main", 1, []call{{id: "c2", model: "m", at: base, frags: 1, in: 2, out: 2}})
+ if _, err := deriver(z, base.Add(time.Hour), metrics.Options{}).Pass(nil); err != nil {
+ t.Fatal(err)
+ }
+ files, err := storage.NewSpool(z).List()
+ if err != nil {
+ t.Fatal(err)
+ }
+ owners := map[string]int{}
+ for _, f := range files {
+ owner, ok := metrics.SpoolOwner(filepath.Base(f.Path))
+ if !ok {
+ t.Fatalf("%s has no owner", f.Path)
+ }
+ owners[owner]++
+ }
+ if len(files) != 2 || owners["abc"] != 1 || owners["abc-1"] != 1 {
+ t.Fatalf("owners %v of %d files, want one file each for abc and abc-1", owners, len(files))
+ }
+}
+
+// A session is derived only once the deriver has read every landed file of
+// it as the file is now. A file landed since is not derived yet.
+func TestDerivedAllNeedsEveryLandedFile(t *testing.T) {
+ z := storage.NewZone(t.TempDir())
+ land(t, z, "s1", "main", 1, []call{{id: "c1", model: "m", at: base, frags: 1, in: 1, out: 1}})
+ land(t, z, "s1", "a1", 2, []call{{id: "c2", model: "m", at: base, frags: 1, in: 1, out: 1}})
+ if ok, err := metrics.DerivedAll(z, "s1", nil); err != nil || ok {
+ t.Fatalf("before any pass: derived=%v err=%v", ok, err)
+ }
+ d := deriver(z, base.Add(time.Hour), metrics.Options{})
+ if _, err := d.Pass(nil); err != nil {
+ t.Fatal(err)
+ }
+ if ok, err := metrics.DerivedAll(z, "s1", nil); err != nil || !ok {
+ t.Fatalf("after a pass: derived=%v err=%v", ok, err)
+ }
+ land(t, z, "s1", "main", 3, []call{{id: "c3", model: "m", at: base.Add(time.Minute), frags: 1, in: 1, out: 1}})
+ if ok, err := metrics.DerivedAll(z, "s1", nil); err != nil || ok {
+ t.Fatalf("a file landed since the pass: derived=%v err=%v", ok, err)
+ }
+ if _, err := d.Pass(nil); err != nil {
+ t.Fatal(err)
+ }
+ if ok, err := metrics.DerivedAll(z, "s1", nil); err != nil || !ok {
+ t.Fatalf("after the next pass: derived=%v err=%v", ok, err)
+ }
+
+ // A different digest is not the file that was derived.
+ loaded, err := metrics.LoadDerived(z)
+ if err != nil {
+ t.Fatal(err)
+ }
+ files, err := storage.LandedFiles(z, "s1")
+ if err != nil {
+ t.Fatal(err)
+ }
+ digest, err := storage.FileDigest(files[0].Path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ rel := relUnder(t, z, files[0].Path)
+ if !loaded.Has(rel, digest) || loaded.Has(rel, strings.Repeat("0", 64)) {
+ t.Fatalf("Has(%s) must hold for its own digest only", rel)
+ }
+}
+
+// Forget drops only the entries whose landed file is gone, and a session's
+// counted calls only once no file of it is left.
+func TestForgetDropsOnlyWhatIsGone(t *testing.T) {
+ z := storage.NewZone(t.TempDir())
+ p1 := land(t, z, "s1", "main", 1, []call{{id: "c1", model: "m", at: base, frags: 1, in: 1, out: 1}})
+ p2 := land(t, z, "s1", "a1", 2, []call{{id: "c2", model: "m", at: base, frags: 1, in: 1, out: 1}})
+ land(t, z, "s2", "main", 1, []call{{id: "c3", model: "m", at: base, frags: 1, in: 1, out: 1}})
+ d := deriver(z, base.Add(time.Hour), metrics.Options{})
+ if _, err := d.Pass(nil); err != nil {
+ t.Fatal(err)
+ }
+ now := base.Add(2 * time.Hour)
+
+ if err := storage.DeleteTree(p2, nil); err != nil {
+ t.Fatal(err)
+ }
+ if err := metrics.Forget(z, []string{"s1"}, now); err != nil {
+ t.Fatal(err)
+ }
+ derived, sessions := readMetricsState(t, z)
+ if _, ok := derived[relUnder(t, z, p1)]; !ok {
+ t.Fatal("the entry of a file that is still there was dropped")
+ }
+ if _, ok := derived[relUnder(t, z, p2)]; ok {
+ t.Fatal("the entry of a file that is gone stayed")
+ }
+ if _, ok := sessions["s1"]; !ok {
+ t.Fatal("s1 still has a landed file, and lost its counted calls")
+ }
+
+ // The session gone whole, as a scenario removal leaves it.
+ if err := storage.DeleteTree(z.SessionDir("s1"), nil); err != nil {
+ t.Fatal(err)
+ }
+ if err := metrics.Forget(z, []string{"s1"}, now); err != nil {
+ t.Fatal(err)
+ }
+ derived, sessions = readMetricsState(t, z)
+ for rel := range derived {
+ if strings.HasPrefix(rel, "s1/") {
+ t.Fatalf("%s stayed after s1 was removed whole", rel)
+ }
+ }
+ if _, ok := sessions["s1"]; ok {
+ t.Fatal("the counted calls of s1 stayed after it was removed whole")
+ }
+ if _, ok := sessions["s2"]; !ok || len(derived) != 1 {
+ t.Fatalf("s2 was touched: derived %v, sessions %v", derived, sessions)
+ }
+
+ // A pass after it never makes the state of s1 again.
+ if _, err := d.Pass(nil); err != nil {
+ t.Fatal(err)
+ }
+ if _, sessions = readMetricsState(t, z); sessions["s1"] != nil {
+ t.Fatal("a pass made the state of a removed session again")
+ }
+
+ z2 := storage.NewZone(t.TempDir())
+ if err := metrics.Forget(z2, []string{"s1"}, now); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := os.Stat(filepath.Join(storage.NewSpool(z2).Dir(), metrics.StateFile)); !os.IsNotExist(err) {
+ t.Fatalf("Forget made a metrics.state in a root that had none: %v", err)
+ }
+}
diff --git a/internal/parse/parse.go b/internal/parse/parse.go
index e923a17..879f9af 100644
--- a/internal/parse/parse.go
+++ b/internal/parse/parse.go
@@ -27,7 +27,10 @@ package parse
import (
"encoding/json"
+ "errors"
"fmt"
+ "io/fs"
+ "os"
"time"
"github.com/apache/skywalking-ai-sessionizer/internal/assemble"
@@ -150,45 +153,32 @@ func Session(z *storage.Zone, opt Options) (*Round, error) {
opt.Now = time.Now
}
- ix, ok, err := index.Load(z.IndexDir(opt.Session), opt.Session)
- if err != nil {
- return nil, err
- }
- if !ok {
- ix, err = rebuild(z, opt)
+ policy := policyFor(opt.IdleGap)
+ chain := sessionflow.OpenChain(z.Root(), opt.Conversation)
+ nothing := &Round{Session: opt.Session, Conversation: opt.Conversation}
+
+ // A session with no landed files and no chain has nothing to parse. Taking
+ // the lock would create the chain's directory, so the parse returns first
+ // and creates nothing. Otherwise a parse that ran after a scenario root
+ // removed a sent session would leave a stub of its chain behind.
+ if _, err := os.Stat(chain.Dir()); errors.Is(err, fs.ErrNotExist) {
+ empty, err := noLandedFiles(z, opt.Session)
if err != nil {
return nil, err
}
- }
-
- // The index must not be read past what it actually covers. Its own state
- // records that; parsing beyond it would describe records the index does not
- // hold.
- ixState, err := storage.LoadIndexState(z.IndexStatePath(opt.Session), opt.Session)
- if err != nil {
- return nil, err
- }
-
- // The watermark is fixed BEFORE assembly and assembly is bounded by it.
- // Deciding afterwards would let a round contain nodes drawn from evidence
- // its own header and input digest do not cover, which a concurrent
- // collector makes likely rather than theoretical.
- through := ixState.IndexedSeq
- if through == 0 {
- for i := range ix.Entries {
- if s := uint64(ix.Entries[i].Seq); s > through {
- through = s
- }
+ if empty {
+ return nothing, nil
}
}
- policy := policyFor(opt.IdleGap)
- chain := sessionflow.OpenChain(z.Root(), opt.Conversation)
-
// Publishing is a read followed by a write - decide the next round from what
// is on disk, then create it. Two builders doing that at once would both
// read round N and both write an N+1, and because the digest is part of the
// filename their files would not even collide.
+ //
+ // The lock is taken before the index is read. A parser that read the index
+ // first could hold an old one while a scenario removal took the session
+ // away, and then publish a round over evidence that is gone.
lock, err := chain.Lock()
if err != nil {
return nil, err
@@ -212,6 +202,49 @@ func Session(z *storage.Zone, opt Options) (*Round, error) {
if view.Session != opt.Session {
return nil, fmt.Errorf("parse: this chain carries session %q, not %q", view.Session, opt.Session)
}
+ } else {
+ // Checked again under the lock. The chain's directory can exist with
+ // no round in it, holding only the lock, and with no landed files
+ // either there is nothing to publish.
+ empty, err := noLandedFiles(z, opt.Session)
+ if err != nil {
+ return nil, err
+ }
+ if empty {
+ return nothing, nil
+ }
+ }
+
+ ix, ok, err := index.Load(z.IndexDir(opt.Session), opt.Session)
+ if err != nil {
+ return nil, err
+ }
+ if !ok {
+ ix, err = rebuild(z, opt)
+ if err != nil {
+ return nil, err
+ }
+ }
+
+ // The index must not be read past what it actually covers. Its own state
+ // records that; parsing beyond it would describe records the index does not
+ // hold.
+ ixState, err := storage.LoadIndexState(z.IndexStatePath(opt.Session), opt.Session)
+ if err != nil {
+ return nil, err
+ }
+
+ // The watermark is fixed BEFORE assembly and assembly is bounded by it.
+ // Deciding afterwards would let a round contain nodes drawn from evidence
+ // its own header and input digest do not cover, which a concurrent
+ // collector makes likely rather than theoretical.
+ through := ixState.IndexedSeq
+ if through == 0 {
+ for i := range ix.Entries {
+ if s := uint64(ix.Entries[i].Seq); s > through {
+ through = s
+ }
+ }
}
out := &Round{
@@ -337,6 +370,15 @@ func Session(z *storage.Zone, opt Options) (*Round, error) {
return out, nil
}
+// noLandedFiles reports whether a session has no landed file at all.
+func noLandedFiles(z *storage.Zone, session string) (bool, error) {
+ files, err := storage.LandedFiles(z, session)
+ if err != nil {
+ return false, err
+ }
+ return len(files) == 0, nil
+}
+
// rebuild reconstructs a missing index from the landed files themselves.
//
// The rebuilt index is written back, so this cost is paid once rather than on
diff --git a/internal/parse/parse_test.go b/internal/parse/parse_test.go
new file mode 100644
index 0000000..175e148
--- /dev/null
+++ b/internal/parse/parse_test.go
@@ -0,0 +1,174 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package parse_test
+
+import (
+ "errors"
+ "io"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/parse"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+ "github.com/apache/skywalking-ai-sessionizer/pkg/sessiondata"
+ "github.com/apache/skywalking-ai-sessionizer/pkg/sessionflow"
+)
+
+const session = "0438c73b-2367-4ed5-9de3-13ef9a17ed01"
+
+// landOne writes one landed file of one record, the least a parse turns
+// into a round.
+func landOne(t *testing.T, z *storage.Zone) {
+ t.Helper()
+ path := filepath.Join(z.StreamDir(session, storage.StreamMain), storage.LandedName("transcript", storage.Stamp(time.Unix(0, 0)), 1))
+ err := storage.WriteAtomic(path, storage.PermLanded, func(w io.Writer) error {
+ sw, err := sessiondata.NewWriter(w, &sessiondata.Header{H: 1, Seq: 1, At: "2026-09-11T00:00:00Z",
+ Kind: sessiondata.KindTranscript, Adapter: "test/0", Dialect: "test/1",
+ Src: "-p/" + session + ".jsonl", Session: session, Stream: storage.StreamMain})
+ if err != nil {
+ return err
+ }
+ if err := sw.Write(&sessiondata.Record{Ord: 1, Sha: "a", Bytes: 1, ID: "u1", From: sessiondata.FromAgent,
+ Time: "2026-09-11T00:00:01Z",
+ Parts: []sessiondata.Part{{Kind: sessiondata.PartText, Text: "hello", State: "available", Bytes: 5}}}); err != nil {
+ return err
+ }
+ return sw.Close()
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+}
+
+func exists(t *testing.T, path string) bool {
+ t.Helper()
+ _, err := os.Lstat(path)
+ if err != nil && !errors.Is(err, fs.ErrNotExist) {
+ t.Fatal(err)
+ }
+ return err == nil
+}
+
+// A parse of a session with nothing landed and no chain creates nothing: no
+// chain directory and no session directory. The same holds when the chain's
+// directory exists holding only its lock.
+func TestAParseOfNothingCreatesNothing(t *testing.T) {
+ root := t.TempDir()
+ z := storage.NewZone(root)
+ chainDir := filepath.Join(root, "_conversations", session)
+ opt := parse.Options{Conversation: session, Session: session}
+
+ r, err := parse.Session(z, opt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if r.Changed() {
+ t.Fatalf("a parse of nothing wrote round %d", r.Number)
+ }
+ if exists(t, chainDir) || exists(t, z.SessionDir(session)) {
+ t.Fatal("a parse of nothing created a directory")
+ }
+
+ lock, err := sessionflow.OpenChain(root, session).Lock()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := lock.Unlock(); err != nil {
+ t.Fatal(err)
+ }
+ if r, err = parse.Session(z, opt); err != nil {
+ t.Fatal(err)
+ }
+ if r.Changed() {
+ t.Fatalf("a parse of a chain with no round and no landed file wrote round %d", r.Number)
+ }
+ items, err := os.ReadDir(chainDir)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(items) != 1 || items[0].Name() != ".lock" {
+ var names []string
+ for _, it := range items {
+ names = append(names, it.Name())
+ }
+ t.Fatalf("the chain holds %v, want only its lock", names)
+ }
+ if exists(t, z.SessionDir(session)) {
+ t.Fatal("a parse of nothing created the session directory")
+ }
+}
+
+// The chain lock comes before the index. A parse that meets a held chain
+// returns at once, and does not rebuild an index it could not use. Once the
+// session and its chain are removed whole, a parse creates nothing again.
+func TestParseTakesTheChainBeforeTheIndex(t *testing.T) {
+ root := t.TempDir()
+ z := storage.NewZone(root)
+ chainDir := filepath.Join(root, "_conversations", session)
+ opt := parse.Options{Conversation: session, Session: session}
+ landOne(t, z)
+ r, err := parse.Session(z, opt)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if r.Number != 1 {
+ t.Fatalf("the first parse wrote round %d, want 1", r.Number)
+ }
+ if err := storage.DeleteTree(z.IndexDir(session), nil); err != nil {
+ t.Fatal(err)
+ }
+
+ lock, err := sessionflow.OpenChain(root, session).Lock()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := parse.Session(z, opt); !errors.Is(err, storage.ErrChainBusy) {
+ t.Fatalf("a parse against a held chain gave %v, want ErrChainBusy", err)
+ }
+ if exists(t, z.IndexDir(session)) {
+ t.Fatal("the parse rebuilt the index before it held the chain")
+ }
+ if err := lock.Unlock(); err != nil {
+ t.Fatal(err)
+ }
+
+ if r, err = parse.Session(z, opt); err != nil {
+ t.Fatal(err)
+ }
+ if r.Changed() || !exists(t, z.IndexDir(session)) {
+ t.Fatalf("after the release: round %d, index rebuilt %v; want the index rebuilt and no new round",
+ r.Number, exists(t, z.IndexDir(session)))
+ }
+
+ // The session and its chain removed whole, as a scenario removal leaves
+ // them.
+ for _, dir := range []string{z.SessionDir(session), chainDir} {
+ if err := storage.DeleteTree(dir, nil); err != nil {
+ t.Fatal(err)
+ }
+ }
+ if r, err = parse.Session(z, opt); err != nil {
+ t.Fatal(err)
+ }
+ if r.Changed() || exists(t, chainDir) || exists(t, z.SessionDir(session)) {
+ t.Fatalf("a parse after the removal wrote round %d or made a directory again", r.Number)
+ }
+}
diff --git a/internal/scenario/build.go b/internal/scenario/build.go
index 1360f76..2bd4601 100644
--- a/internal/scenario/build.go
+++ b/internal/scenario/build.go
@@ -24,6 +24,8 @@ import (
"path/filepath"
"strings"
"time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
)
// Format names a writer: what a build leaves in the output directory.
@@ -48,6 +50,9 @@ type Built struct {
Config string
Files []string
Events int
+ // Marker is the marker a claude-code build wrote for the session, and
+ // empty for an sd build, which writes none.
+ Marker string
}
// Build plans a scenario and writes it into out with the named writer, then
@@ -78,7 +83,24 @@ func Build(sc *Scenario, format Format, out string, opts Options) (*Built, error
res := &Built{Plan: p, Session: p.Session, Format: format, Out: out, Events: len(p.Events)}
switch format {
case FormatClaudeCode:
- res.Files, err = writeClaudeCode(p, source)
+ // The sign of a scenario root. A pipeline over a root that has it
+ // takes the scenario lock, and only such a pipeline removes.
+ if err := os.MkdirAll(filepath.Join(out, storage.ScenarioDir), 0o755); err != nil {
+ return nil, err
+ }
+ if err := clearMarker(out, source, p.Session); err != nil {
+ return nil, err
+ }
+ var marked []MarkerFile
+ res.Files, marked, err = writeClaudeCode(p, source)
+ if err != nil {
+ return nil, err
+ }
+ // Written last, after every file of the session. The writer writes
+ // one file after another with no other sign of being done, so the
+ // marker is the only proof that the build finished the session.
+ res.Marker = MarkerPath(source, p.Session)
+ err = WriteMarker(res.Marker, newMarker(p.Session, opts.Remove, marked))
case FormatSD:
now := opts.At
if now.IsZero() {
diff --git a/internal/scenario/expect/expect.go b/internal/scenario/expect/expect.go
index c18bed3..a39beba 100644
--- a/internal/scenario/expect/expect.go
+++ b/internal/scenario/expect/expect.go
@@ -87,6 +87,12 @@ type Properties struct {
// step, never a step. Checked when a scenario has changes.
ChangesLeaveTheFold *bool `yaml:"changes_leave_the_fold"`
MetricsMatchThePlan *bool `yaml:"metrics_match_the_plan"`
+ // RemovedAfterSent says what a pipeline's removal does with the session,
+ // on copies of the finished root. Nothing goes before all of it is sent
+ // to the one receiver the pipeline sends to. All of it goes once it is,
+ // by its marker's policy, and nothing is made or sent again afterwards.
+ // A session with no marker stays.
+ RemovedAfterSent *bool `yaml:"removed_after_sent"`
}
// On reports whether a property is enabled.
diff --git a/internal/scenario/marker.go b/internal/scenario/marker.go
new file mode 100644
index 0000000..d9f36fa
--- /dev/null
+++ b/internal/scenario/marker.go
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scenario
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "io/fs"
+ "os"
+ "path"
+ "path/filepath"
+ "regexp"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// MarkerDir is the directory, under a build's source directory, that holds
+// one marker for each session a claude-code build wrote.
+//
+// Discovery reads it as a project directory. It holds only .json files,
+// which are neither transcripts nor session directories, so nothing in it is
+// ever collected.
+const MarkerDir = ".asz-scenario"
+
+// The policies a marker names: when a pipeline may remove a session once all
+// of it has been sent.
+const (
+ PolicyImmediately = "immediately"
+ PolicyRetain = "retain"
+)
+
+// The states a marker is in.
+const (
+ // MarkerWritten is how the build leaves it.
+ MarkerWritten = "written"
+ // MarkerRemoving says a pipeline has started to remove the session. From
+ // then on a pipeline finishes the removal without deciding again, and the
+ // build refuses to write the session.
+ MarkerRemoving = "removing"
+)
+
+// markerSchema is the one schema a reader takes.
+const markerSchema = 1
+
+// Marker is what a claude-code build records about one session it wrote: the
+// policy for its removal, and every file it wrote with the size and digest
+// the file had.
+//
+// The build writes it after every file of the session. So a marker is proof
+// that the build finished writing the session, and the files it lists are
+// the only ones a removal may delete. A session with no marker, which is every
+// real Claude Code session, is never removed.
+type Marker struct {
+ Schema int `json:"schema"`
+ Session string `json:"session"`
+ Policy string `json:"policy"`
+ Retain string `json:"retain,omitempty"`
+ State string `json:"state"`
+ Files []MarkerFile `json:"files"`
+}
+
+// MarkerFile is one file the build wrote for a session. Path is relative to
+// the build's source directory, with forward slashes.
+type MarkerFile struct {
+ Path string `json:"path"`
+ Size int64 `json:"size"`
+ SHA256 string `json:"sha256"`
+}
+
+// Removal is when a pipeline may remove a session once all of it is sent: at
+// once when Retain is zero, or once the session's last record is Retain old.
+type Removal struct{ Retain time.Duration }
+
+// Immediate reports whether the session goes as soon as it is sent.
+func (r Removal) Immediate() bool { return r.Retain <= 0 }
+
+// String is the policy the way a person writes it after --remove.
+func (r Removal) String() string {
+ if r.Immediate() {
+ return PolicyImmediately
+ }
+ return shortDuration(r.Retain)
+}
+
+// Describe says the policy in the words the build prints.
+func (r Removal) Describe() string {
+ if r.Immediate() {
+ return "immediately, once sent"
+ }
+ return shortDuration(r.Retain) + " after its last record, once sent"
+}
+
+// shortDuration drops the zero minutes and seconds Go prints, so 24h0m0s
+// reads 24h.
+func shortDuration(d time.Duration) string {
+ s := d.String()
+ if strings.HasSuffix(s, "m0s") {
+ s = strings.TrimSuffix(s, "0s")
+ }
+ if strings.HasSuffix(s, "h0m") {
+ s = strings.TrimSuffix(s, "0m")
+ }
+ return s
+}
+
+// ParseRemove reads a removal policy: immediately, a Go duration such as 30m
+// or 24h, or a whole number of days such as 7d, as metrics_lookback takes
+// them. Zero, a negative value and anything else are refused, because a
+// retention of nothing is immediately said another way.
+func ParseRemove(s string) (Removal, error) {
+ s = strings.TrimSpace(s)
+ if s == PolicyImmediately {
+ return Removal{}, nil
+ }
+ var d time.Duration
+ if days, ok := strings.CutSuffix(s, "d"); ok {
+ n, err := strconv.Atoi(days)
+ if err != nil {
+ return Removal{}, fmt.Errorf("scenario: %q is not a removal policy", s)
+ }
+ d = time.Duration(n) * 24 * time.Hour
+ } else {
+ var err error
+ if d, err = time.ParseDuration(s); err != nil {
+ return Removal{}, fmt.Errorf("scenario: %q is not a removal policy", s)
+ }
+ }
+ if d <= 0 {
+ return Removal{}, fmt.Errorf("scenario: %q is not a removal policy; a retention must be longer than zero", s)
+ }
+ return Removal{Retain: d}, nil
+}
+
+// MarkerPath is where the marker of a session is, under a build's source
+// directory.
+func MarkerPath(source, session string) string {
+ return filepath.Join(source, MarkerDir, session+".json")
+}
+
+// IsMarkerName reports whether a file in MarkerDir is a marker. A file that
+// WriteAtomic is still writing starts with .tmp- and is not one.
+func IsMarkerName(name string) bool {
+ return strings.HasSuffix(name, ".json") && !strings.HasPrefix(name, ".tmp-")
+}
+
+var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`)
+
+// ReadMarker reads a marker and checks everything a removal relies on. The
+// error names the marker and what is wrong with it.
+func ReadMarker(p string) (*Marker, error) {
+ data, err := os.ReadFile(p)
+ if err != nil {
+ return nil, fmt.Errorf("marker %s: %w", p, err)
+ }
+ var m Marker
+ if err := json.Unmarshal(data, &m); err != nil {
+ return nil, fmt.Errorf("marker %s: not a marker: %w", p, err)
+ }
+ bad := func(format string, args ...any) (*Marker, error) {
+ return nil, fmt.Errorf("marker %s: %s", p, fmt.Sprintf(format, args...))
+ }
+ if m.Schema != markerSchema {
+ return bad("the schema is %d, and this asz reads %d", m.Schema, markerSchema)
+ }
+ name := strings.TrimSuffix(filepath.Base(p), ".json")
+ if m.Session != name {
+ return bad("it names the session %q, not %q", m.Session, name)
+ }
+ if !claudecode.IsSessionID(m.Session) {
+ return bad("%q is not a session id discovery would find", m.Session)
+ }
+ if _, err := m.Removal(); err != nil {
+ return bad("%v", err)
+ }
+ if m.State != MarkerWritten && m.State != MarkerRemoving {
+ return bad("the state %q is not %s or %s", m.State, MarkerWritten, MarkerRemoving)
+ }
+ seen := map[string]bool{}
+ for _, f := range m.Files {
+ // A path that leaves the source directory would let a marker name a
+ // file the build never wrote, such as a real transcript.
+ if f.Path == "" || strings.Contains(f.Path, `\`) || path.Clean(f.Path) != f.Path || !filepath.IsLocal(filepath.FromSlash(f.Path)) {
+ return bad("the path %q is not inside the source directory", f.Path)
+ }
+ if seen[f.Path] {
+ return bad("the path %q is listed twice", f.Path)
+ }
+ seen[f.Path] = true
+ if f.Size < 0 || !hexDigest.MatchString(f.SHA256) {
+ return bad("the path %q has no size or no SHA-256", f.Path)
+ }
+ }
+ return &m, nil
+}
+
+// WriteMarker writes a marker in one rename. It is not read-only: a pipeline
+// rewrites it when it starts a removal, and on Windows a rename cannot
+// replace a read-only file.
+func WriteMarker(p string, m *Marker) error {
+ data, err := json.MarshalIndent(m, "", " ")
+ if err != nil {
+ return err
+ }
+ return storage.WriteAtomic(p, storage.PermState, func(w io.Writer) error {
+ _, err := w.Write(append(data, '\n'))
+ return err
+ })
+}
+
+// Removal is the policy the marker names.
+func (m *Marker) Removal() (Removal, error) {
+ switch m.Policy {
+ case PolicyImmediately:
+ return Removal{}, nil
+ case PolicyRetain:
+ d, err := time.ParseDuration(m.Retain)
+ if err != nil || d <= 0 {
+ return Removal{}, fmt.Errorf("the retention %q is not a duration longer than zero", m.Retain)
+ }
+ return Removal{Retain: d}, nil
+ }
+ return Removal{}, fmt.Errorf("the policy %q is not %s or %s", m.Policy, PolicyImmediately, PolicyRetain)
+}
+
+// newMarker is the marker a build writes for a session.
+func newMarker(session string, r Removal, files []MarkerFile) *Marker {
+ m := &Marker{Schema: markerSchema, Session: session, Policy: PolicyImmediately, State: MarkerWritten, Files: files}
+ if !r.Immediate() {
+ m.Policy, m.Retain = PolicyRetain, r.Retain.String()
+ }
+ if m.Files == nil {
+ m.Files = []MarkerFile{}
+ }
+ return m
+}
+
+// clearMarker gets a session ready to be written again. A marker that says
+// removing means a pipeline is deleting the session's files right now, so
+// the build stops rather than write under it. Any other marker is deleted
+// first: a pipeline that meets no marker never removes, so an earlier marker
+// cannot vouch for files this build is about to rewrite.
+func clearMarker(out, source, session string) error {
+ p := MarkerPath(source, session)
+ data, err := os.ReadFile(p)
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ var old struct {
+ State string `json:"state"`
+ }
+ if err := json.Unmarshal(data, &old); err != nil {
+ return fmt.Errorf("scenario: %s is not a marker this build can read (%w); delete it once no pipeline over %s is removing session %s, or use another --out", p, err, out, session)
+ }
+ if old.State == MarkerRemoving {
+ return fmt.Errorf("scenario: session %s is being removed by a pipeline over %s; build it again once that has finished, or use another --out", session, out)
+ }
+ return os.Remove(p)
+}
diff --git a/internal/scenario/marker_test.go b/internal/scenario/marker_test.go
new file mode 100644
index 0000000..fe01165
--- /dev/null
+++ b/internal/scenario/marker_test.go
@@ -0,0 +1,348 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package scenario
+
+import (
+ "bytes"
+ "crypto/sha256"
+ "encoding/hex"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+)
+
+// everyKindOfFile is one session holding every kind of file a removal
+// deletes: a main transcript, a child with a meta file, a workflow whose
+// script is filed under another project, and plugin output.
+const everyKindOfFile = `title: every kind of file a removal deletes
+steps:
+ - input: fix the timeout and check everything
+ - call:
+ text: Fixing it.
+ tool:
+ name: Bash
+ input: {command: "sed -i 's/= 10/= 30/' server.go"}
+ result: {text: "", after: 500ms}
+ changes:
+ - {path: server.go, before: "package main\n\nvar timeout = 10\n", after: "package main\n\nvar timeout = 30\n"}
+ usage: {in: 2, out: 50, cache_read: 900, cache_write: 100}
+ - call:
+ agent: {name: helper, prompt: look at the tests, steps: [{call: {text: tests look fine, usage: {out: 20}}}], notify: true}
+ - call:
+ workflow:
+ name: verify
+ script_project: -Users-dev-script-filed-elsewhere
+ children:
+ - {name: lint, prompt: run the linter, steps: [{call: {text: lint done}}]}
+ - call: {text: Everything checks out.}
+`
+
+var markerAt = time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+
+func loadEveryKind(t *testing.T) *Scenario {
+ t.Helper()
+ path := filepath.Join(t.TempDir(), "every-kind.yaml")
+ if err := os.WriteFile(path, []byte(everyKindOfFile), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ sc, err := Load(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return sc
+}
+
+// TestTheMarkerListsEveryFileOfTheSessionAndNothingElse. A removal deletes
+// exactly what the marker lists, so a file left out would outlive its
+// session, and a file listed that is not the session's would be deleted with
+// it. The two noise files are shared by every session of the project.
+func TestTheMarkerListsEveryFileOfTheSessionAndNothingElse(t *testing.T) {
+ sc := loadEveryKind(t)
+ out := t.TempDir()
+ b, err := Build(sc, FormatClaudeCode, out, Options{At: markerAt, Remove: Removal{Retain: 24 * time.Hour}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if fi, err := os.Stat(filepath.Join(b.Out, "_scenario")); err != nil || !fi.IsDir() {
+ t.Fatalf("a claude-code build left no _scenario directory in its root: %v", err)
+ }
+ source := filepath.Join(b.Out, "_source")
+ if b.Marker != MarkerPath(source, b.Session) {
+ t.Fatalf("the marker is at %s, want %s", b.Marker, MarkerPath(source, b.Session))
+ }
+ m, err := ReadMarker(b.Marker)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if m.Policy != PolicyRetain || m.Retain != "24h0m0s" || m.State != MarkerWritten || m.Session != b.Session {
+ t.Fatalf("the marker says policy %q, retain %q, state %q, session %q", m.Policy, m.Retain, m.State, m.Session)
+ }
+
+ onDisk := map[string]bool{}
+ err = filepath.WalkDir(source, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ if d.IsDir() {
+ if d.Name() == MarkerDir {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ rel, _ := filepath.Rel(source, p)
+ rel = filepath.ToSlash(rel)
+ if strings.HasSuffix(rel, "/not-a-uuid.jsonl") || strings.Contains(rel, "/memory/") {
+ return nil
+ }
+ onDisk[rel] = true
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ listed := map[string]bool{}
+ for _, f := range m.Files {
+ listed[f.Path] = true
+ data, err := os.ReadFile(filepath.Join(source, filepath.FromSlash(f.Path)))
+ if err != nil {
+ t.Fatalf("the marker lists %s: %v", f.Path, err)
+ }
+ sum := sha256.Sum256(data)
+ if int64(len(data)) != f.Size || hex.EncodeToString(sum[:]) != f.SHA256 {
+ t.Errorf("%s: the marker records %d bytes and %s, the file has %d bytes", f.Path, f.Size, f.SHA256, len(data))
+ }
+ }
+ for p := range onDisk {
+ if !listed[p] {
+ t.Errorf("the build wrote %s and the marker does not list it", p)
+ }
+ }
+ for p := range listed {
+ if !onDisk[p] {
+ t.Errorf("the marker lists %s, which is not a file of the session", p)
+ }
+ }
+ // Every kind is there, so the comparison above covers each of them.
+ for _, part := range []string{b.Session + ".jsonl", ".meta.json", "journal.jsonl", "-Users-dev-script-filed-elsewhere/", PluginOutputDir + "/"} {
+ found := false
+ for p := range listed {
+ found = found || strings.Contains(p, part)
+ }
+ if !found {
+ t.Errorf("the marker lists no file with %q in its path", part)
+ }
+ }
+ // What the command prints does not change: the noise is still written
+ // and listed, and the marker is not one of the files.
+ noise := 0
+ for _, f := range b.Files {
+ if strings.HasSuffix(f, "not-a-uuid.jsonl") || strings.HasSuffix(f, "notes.md") {
+ noise++
+ }
+ if strings.Contains(f, MarkerDir) {
+ t.Errorf("the build lists its marker %s among the files it wrote", f)
+ }
+ }
+ if noise != 2 {
+ t.Errorf("the build lists %d noise files, want 2", noise)
+ }
+}
+
+// TestABuildLeavesASessionBeingRemovedAlone. A pipeline that started a
+// removal deletes the files its marker lists. A build that wrote them again
+// meanwhile would have its new files deleted, or stop the removal half way.
+func TestABuildLeavesASessionBeingRemovedAlone(t *testing.T) {
+ sc, err := Load(filepath.Join("..", "..", "tests", "scenarios", "assembly.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ out := t.TempDir()
+ b, err := Build(sc, FormatClaudeCode, out, Options{At: markerAt})
+ if err != nil {
+ t.Fatal(err)
+ }
+ m, err := ReadMarker(b.Marker)
+ if err != nil {
+ t.Fatal(err)
+ }
+ m.State = MarkerRemoving
+ if err := WriteMarker(b.Marker, m); err != nil {
+ t.Fatal(err)
+ }
+ main := filepath.Join(b.Out, "_source", filepath.FromSlash(m.Files[0].Path))
+ before, err := os.ReadFile(main)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, err = Build(sc, FormatClaudeCode, out, Options{At: markerAt.Add(time.Hour)})
+ if err == nil || !strings.Contains(err.Error(), "is being removed") {
+ t.Fatalf("a build over a session being removed gave %v", err)
+ }
+ after, err := os.ReadFile(main)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !bytes.Equal(before, after) {
+ t.Fatal("the refused build still rewrote the session's files")
+ }
+}
+
+// TestABuildReplacesAnEarlierMarker. A marker vouches for the bytes it
+// records. One left from an earlier build must not survive a build that
+// writes the files again, or it would describe files that are no longer
+// there, and keep a policy nobody asked for this time.
+func TestABuildReplacesAnEarlierMarker(t *testing.T) {
+ sc, err := Load(filepath.Join("..", "..", "tests", "scenarios", "assembly.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ out := t.TempDir()
+ b, err := Build(sc, FormatClaudeCode, out, Options{At: markerAt})
+ if err != nil {
+ t.Fatal(err)
+ }
+ m, err := ReadMarker(b.Marker)
+ if err != nil {
+ t.Fatal(err)
+ }
+ m.Files[0].SHA256 = strings.Repeat("0", 64)
+ m.Policy, m.Retain = PolicyRetain, "1h0m0s"
+ if err := WriteMarker(b.Marker, m); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := Build(sc, FormatClaudeCode, out, Options{At: markerAt}); err != nil {
+ t.Fatal(err)
+ }
+ again, err := ReadMarker(b.Marker)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if again.Policy != PolicyImmediately || again.Retain != "" {
+ t.Fatalf("the policy of the earlier marker survived: %q %q", again.Policy, again.Retain)
+ }
+ data, err := os.ReadFile(filepath.Join(b.Out, "_source", filepath.FromSlash(again.Files[0].Path)))
+ if err != nil {
+ t.Fatal(err)
+ }
+ sum := sha256.Sum256(data)
+ if again.Files[0].SHA256 != hex.EncodeToString(sum[:]) {
+ t.Fatal("the marker still records a digest the file does not have")
+ }
+}
+
+// TestAnSDBuildWritesNoMarker. An sd build writes no source, so there is
+// nothing a removal could delete, and its root is not a scenario root.
+func TestAnSDBuildWritesNoMarker(t *testing.T) {
+ sc, err := Load(filepath.Join("..", "..", "tests", "scenarios", "assembly.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ out := t.TempDir()
+ b, err := Build(sc, FormatSD, out, Options{At: markerAt})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if b.Marker != "" {
+ t.Fatalf("an sd build reports the marker %s", b.Marker)
+ }
+ for _, p := range []string{filepath.Join(b.Out, "_scenario"), filepath.Join(b.Out, "_source", MarkerDir)} {
+ if _, err := os.Stat(p); !os.IsNotExist(err) {
+ t.Errorf("an sd build made %s", p)
+ }
+ }
+}
+
+// TestParseRemove holds the syntax of --remove. A retention of zero or less
+// is refused rather than read as immediately, since a person who wrote a
+// duration meant one.
+func TestParseRemove(t *testing.T) {
+ for s, want := range map[string]time.Duration{"immediately": 0, "30m": 30 * time.Minute, "24h": 24 * time.Hour, "7d": 7 * 24 * time.Hour} {
+ got, err := ParseRemove(s)
+ if err != nil || got.Retain != want {
+ t.Errorf("ParseRemove(%q) = %v, %v; want %s", s, got.Retain, err, want)
+ }
+ }
+ for _, s := range []string{"0", "0d", "-1h", "soon", "", "1.5d"} {
+ if _, err := ParseRemove(s); err == nil {
+ t.Errorf("ParseRemove(%q) was accepted", s)
+ }
+ }
+ if got := (Removal{Retain: 24 * time.Hour}).Describe(); got != "24h after its last record, once sent" {
+ t.Errorf("Describe gave %q", got)
+ }
+ if got := (Removal{}).Describe(); got != "immediately, once sent" {
+ t.Errorf("Describe gave %q", got)
+ }
+}
+
+// TestReadMarkerRefusesWhatARemovalCannotTrust. Each field decides what a
+// removal deletes or when, so a marker that is wrong in any of them keeps its
+// session instead.
+func TestReadMarkerRefusesWhatARemovalCannotTrust(t *testing.T) {
+ const id = "c30736f2-ac0c-4a72-89b1-01a0844af62d"
+ empty := sha256.Sum256(nil)
+ good := func() *Marker {
+ return &Marker{Schema: 1, Session: id, Policy: PolicyImmediately, State: MarkerWritten,
+ Files: []MarkerFile{{Path: "-Users-dev-scenario/" + id + ".jsonl", SHA256: hex.EncodeToString(empty[:])}}}
+ }
+ dir := t.TempDir()
+ if _, err := readMarkerAt(t, dir, id+".json", good()); err != nil {
+ t.Fatalf("a good marker was refused: %v", err)
+ }
+ for i, c := range []struct {
+ name, file string
+ change func(*Marker)
+ want string
+ }{
+ {"a path out of the source", id + ".json", func(m *Marker) { m.Files[0].Path = "../elsewhere/x.jsonl" }, "not inside the source directory"},
+ {"an absolute path", id + ".json", func(m *Marker) { m.Files[0].Path = "/etc/passwd" }, "not inside the source directory"},
+ {"another session than its name", id + ".json", func(m *Marker) { m.Session = "aaaaaaaa-bbbb-4ccc-8ddd-000000000001" }, "names the session"},
+ {"a name that is no session id", "notes.json", func(m *Marker) { m.Session = "notes" }, "not a session id"},
+ {"an unknown policy", id + ".json", func(m *Marker) { m.Policy = "later" }, "policy"},
+ {"a retention of nothing", id + ".json", func(m *Marker) { m.Policy, m.Retain = PolicyRetain, "0s" }, "retention"},
+ {"an unknown state", id + ".json", func(m *Marker) { m.State = "done" }, "state"},
+ {"a path listed twice", id + ".json", func(m *Marker) { m.Files = append(m.Files, m.Files[0]) }, "listed twice"},
+ {"another schema", id + ".json", func(m *Marker) { m.Schema = 2 }, "schema"},
+ } {
+ m := good()
+ c.change(m)
+ _, err := readMarkerAt(t, filepath.Join(dir, strings.Repeat("x", i+1)), c.file, m)
+ if err == nil || !strings.Contains(err.Error(), c.want) {
+ t.Errorf("%s: ReadMarker gave %v, want an error saying %q", c.name, err, c.want)
+ }
+ }
+ for name, want := range map[string]bool{id + ".json": true, ".tmp-123456": false, id + ".json.tmp": false, "notes.md": false} {
+ if IsMarkerName(name) != want {
+ t.Errorf("IsMarkerName(%q) = %v", name, !want)
+ }
+ }
+}
+
+// readMarkerAt writes a marker under dir with the file name given and reads
+// it back.
+func readMarkerAt(t *testing.T, dir, name string, m *Marker) (*Marker, error) {
+ t.Helper()
+ p := filepath.Join(dir, name)
+ if err := WriteMarker(p, m); err != nil {
+ t.Fatal(err)
+ }
+ return ReadMarker(p)
+}
diff --git a/internal/scenario/plan.go b/internal/scenario/plan.go
index bb9d5bc..6e634cf 100644
--- a/internal/scenario/plan.go
+++ b/internal/scenario/plan.go
@@ -41,6 +41,10 @@ type Options struct {
// which is what a build that keeps feeding the root needs beside it.
// It has no effect on the plan.
Watch bool
+ // Remove is when a pipeline may remove a session a claude-code build
+ // wrote, once all of it is sent. The build records it in the session's
+ // marker. It has no effect on the plan.
+ Remove Removal
}
// EventKind is one shape of record.
diff --git a/internal/scenario/remove/remove.go b/internal/scenario/remove/remove.go
new file mode 100644
index 0000000..017cf30
--- /dev/null
+++ b/internal/scenario/remove/remove.go
@@ -0,0 +1,1282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package remove removes a scenario session once all of it has been sent.
+//
+// Only a session a claude-code scenario build marked is ever removed, and
+// only by the policy its marker names. A real Claude Code session has no
+// marker, so nothing here ever touches it. The product configuration has no
+// removal setting at all.
+//
+// It is collector side. It reads the source directory, the storage root and
+// the export state, and it imports nothing that assembles or serves.
+package remove
+
+import (
+ "bufio"
+ "bytes"
+ "crypto/sha256"
+ "encoding/hex"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "regexp"
+ "sort"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecodechanges"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
+ "github.com/apache/skywalking-ai-sessionizer/internal/metrics"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+ "github.com/apache/skywalking-ai-sessionizer/pkg/sessiondata"
+)
+
+// MaxPerPass bounds how many sessions one pass takes through step 0, resumed
+// ones included. A removal holds the export lock, and asz push waits two
+// minutes for it. The time one removal takes was not measured, so the cap
+// bounds the hold whatever it turns out to be. It is a starting value.
+const MaxPerPass = 16
+
+// conversationsDir is where the chains are, under the storage root. It is
+// written out because the chain's package is server side.
+const conversationsDir = "_conversations"
+
+// changesPrefix is where, under the local adapter's source directory, the
+// changes adapter's source directory is. The build writes the plugin output
+// there, and a removal checks that the changes adapter reads it there.
+const changesPrefix = "plugins/data/"
+
+// Remover removes the sessions a scenario build marked, once all of each is
+// sent. The pipeline calls Pass at the end of every pass, after the push,
+// while it holds the scenario lock.
+type Remover struct {
+ Zone *storage.Zone
+ // Source is claude-code-local's source directory, where the build wrote
+ // the session and its marker.
+ Source string
+ Local *claudecode.Collector
+ LocalMatch func(claudecode.Session) bool
+ // Changes is nil when the changes adapter is not enabled.
+ Changes *claudecodechanges.Collector
+ ChangesRoot string
+ ChangesMatch func(claudecodechanges.Session) bool
+ // Derives says claude-code-local derives metrics, so every landed file
+ // must be derived before its session can go.
+ Derives bool
+ Now func() time.Time
+ // Hook is called at every point a removal reaches, in order. An error
+ // stops the removal there, as a crash would, and the locks are released
+ // as a dying process's are. Tests use it to run the first steps only.
+ Hook func(Point) error
+
+ // said holds the blocked reasons already reported, by session, so each
+ // is printed once per process. The empty session is every session.
+ said map[string]map[string]bool
+ // stopped holds sessions whose removal stopped on a check, left for a
+ // person. They are not tried again by this process.
+ stopped map[string]bool
+ // last is the latest record time found in each session's landed files.
+ // It only grows while a session is kept, since landed files are never
+ // changed, so a session it shows as too young is still too young.
+ last map[string]time.Time
+}
+
+// Input is what the pipeline knows that the root does not say.
+type Input struct {
+ // Endpoint names the receiver the pipeline sends to, as otlp.EndpointOf
+ // gives it. Empty when there is no receiver.
+ Endpoint string
+ SendLogs bool
+ SendMetrics bool
+ // Busy holds sessions another process held this pass, the pipeline's
+ // sessions to parse again.
+ Busy map[string]bool
+}
+
+// Result is what one pass did.
+type Result struct {
+ Removed []string
+ // Kept holds the reasons a session needs a person that this process has
+ // not reported yet. The pipeline prints each once.
+ Kept []Kept
+ // Waiting holds why a session is not ready yet. Nothing prints these.
+ // They change on their own, and tests read them.
+ Waiting []Kept
+ // DeriveNext holds sessions whose only missing condition is that metrics
+ // were not derived from all of their landed files. The pipeline adds them
+ // to the next derivation, because a session that no longer lands anything
+ // is otherwise not derived again.
+ DeriveNext []string
+ Errors []error
+}
+
+// Kept is a session kept this pass, and why. An empty Session means every
+// marked session.
+type Kept struct{ Session, Reason string }
+
+// Point is a place a removal reaches. The steps are 0 to 7:
+//
+// {0, done} the marker says removing
+// {1, file, k} ... {1, dirs} after each source file, then its directories
+// {2, unlocked} {2, renamed} {2, deleting} {2, done} <root>/S
+// {3, unlocked} {3, renamed} {3, deleting} {3, done} <root>/_conversations/S
+// {4, file, k} ... {4, done} after each spool file
+// {5, done} {6, done} push.state and metrics.state, once per pass
+// {7, done} the marker is deleted
+//
+// Steps 5 and 6 run once for every session that finished step 4 in the
+// pass, so their Session is empty. deleting comes after the first entry of
+// the renamed tree is deleted, and only when it has one.
+type Point struct {
+ Step int
+ Where string
+ Session string
+ // N counts the files inside step 1 or step 4, from 1.
+ N int
+}
+
+// The places inside a step.
+const (
+ WhereDone = "done"
+ WhereFile = "file"
+ WhereDirs = "dirs"
+ WhereUnlocked = "unlocked"
+ WhereRenamed = "renamed"
+ WhereDeleting = "deleting"
+)
+
+// crash is what a Hook returned. Nothing after it runs.
+type crash struct{ err error }
+
+func (c crash) Error() string { return "stopped by the hook: " + c.err.Error() }
+func (c crash) Unwrap() error { return c.err }
+
+// stop is a check inside a removal that refused. The session is left for a
+// person, and its marker keeps saying removing.
+type stop struct{ path, why string }
+
+func (s stop) Error() string { return s.path + " " + s.why }
+
+// verdict is the answer to one condition. An empty reason means it holds.
+type verdict struct {
+ reason string
+ // blocked means a person has to act. Otherwise the session waits.
+ blocked bool
+ // derive means the reason is only that metrics were not derived yet.
+ derive bool
+ // every means the reason applies to every marked session, and the pass
+ // stops deciding.
+ every bool
+}
+
+func waiting(format string, a ...any) verdict { return verdict{reason: fmt.Sprintf(format, a...)} }
+func blocked(format string, a ...any) verdict {
+ return verdict{reason: fmt.Sprintf(format, a...), blocked: true}
+}
+
+// Pass removes every marked session that is sent and that its policy lets
+// go, up to MaxPerPass, and finishes any removal an earlier pass left.
+func (r *Remover) Pass(in Input) *Result {
+ res := &Result{}
+ if r.said == nil {
+ r.said, r.stopped, r.last = map[string]map[string]bool{}, map[string]bool{}, map[string]time.Time{}
+ }
+ root := r.Zone.Root()
+ // A directory an earlier pass renamed away and did not finish deleting.
+ // Nothing lists it, so it can wait, but nothing else deletes it either.
+ if err := storage.SweepRemoved(root); errors.Is(err, storage.ErrUnsafeRemoved) {
+ // A link at _removed leads out of the root. The storage functions
+ // refuse it. The pass stops here too, before any session reaches
+ // step 0. Otherwise step 1 would delete a session's source, step 2
+ // would be refused, and the session would stay half removed until a
+ // person replaced _removed.
+ r.keep(res, "", fmt.Sprintf("%s is a symbolic link or not a directory; nothing is deleted or moved through it",
+ filepath.Join(root, storage.RemovedDir)))
+ return res
+ } else if err != nil {
+ res.Errors = append(res.Errors, fmt.Errorf("remove: %w", err))
+ }
+ names, err := r.markerNames()
+ if err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ if len(names) == 0 {
+ return res
+ }
+ if reason := r.Blocked(in); reason != "" {
+ r.keep(res, "", reason)
+ return res
+ }
+ lock, err := storage.LockExport(root)
+ if errors.Is(err, storage.ErrExportBusy) {
+ res.Waiting = append(res.Waiting, Kept{Reason: "another pusher holds the export state"})
+ return res
+ }
+ if err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ defer func() { _ = lock.Unlock() }()
+
+ p := &pass{r: r, in: in, root: root, now: r.now()}
+ if p.sent, err = otlp.LoadSent(root); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ switch eps := p.sent.Endpoints(); {
+ case len(eps) == 0:
+ res.Waiting = append(res.Waiting, Kept{Reason: "not sent yet"})
+ return res
+ case len(eps) != 1 || eps[0] != in.Endpoint:
+ r.keep(res, "", fmt.Sprintf("push.state records sending to %s; this pipeline sends to %s; which file reached which receiver cannot be told",
+ strings.Join(eps, ", "), in.Endpoint))
+ return res
+ }
+ if p.derived, err = metrics.LoadDerived(r.Zone); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ if p.spool, err = spoolOwners(root); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+
+ var finished []*session
+ reached := 0
+ for _, c := range r.candidates(names) {
+ if reached >= MaxPerPass {
+ break
+ }
+ if r.stopped[c.id] {
+ continue
+ }
+ if c.err != nil {
+ r.keep(res, c.id, c.err.Error())
+ continue
+ }
+ s := &session{id: c.id, markerPath: c.path, m: c.m}
+ var v verdict
+ if c.m.State == scenario.MarkerRemoving {
+ // Decided by an earlier pass, which proved everything then. Only
+ // the locks are needed to go on.
+ v = p.lock(s)
+ } else {
+ v = p.decide(s)
+ }
+ if v.reason != "" {
+ s.unlock()
+ switch {
+ case v.every:
+ r.keep(res, "", v.reason)
+ case v.blocked:
+ r.keep(res, s.id, v.reason)
+ default:
+ res.Waiting = append(res.Waiting, Kept{Session: s.id, Reason: v.reason})
+ }
+ if v.derive {
+ res.DeriveNext = append(res.DeriveNext, s.id)
+ }
+ if v.every {
+ break
+ }
+ continue
+ }
+ reached++
+ err := r.run(s)
+ s.unlock()
+ if err != nil {
+ var c crash
+ var st stop
+ switch {
+ case errors.As(err, &c):
+ res.Errors = append(res.Errors, fmt.Errorf("%s: %w", s.id, err))
+ return res
+ case errors.As(err, &st):
+ r.stopped[s.id] = true
+ res.Errors = append(res.Errors, fmt.Errorf("%s: the removal stopped: %w; the session is left for a person", s.id, st))
+ default:
+ res.Errors = append(res.Errors, fmt.Errorf("%s: the removal did not finish, and the next pass goes on with it: %w", s.id, err))
+ }
+ continue
+ }
+ finished = append(finished, s)
+ }
+ if len(finished) == 0 {
+ return res
+ }
+
+ // Steps 5 and 6, once for all of them. Each rewrites a whole state file,
+ // so one write serves every session this pass removed. A line is dropped
+ // only once its file is gone, so no file is ever sent or derived twice.
+ ids := make([]string, 0, len(finished))
+ owned := map[string]bool{}
+ for _, s := range finished {
+ ids = append(ids, s.id)
+ owned[s.id] = true
+ }
+ if err := otlp.ForgetGone(root, ownedBy(owned), p.now); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ if err := r.hook(Point{Step: 5, Where: WhereDone}); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ if err := metrics.Forget(r.Zone, ids, p.now); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ if err := r.hook(Point{Step: 6, Where: WhereDone}); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ // Step 7. The marker goes last, so a session whose removal began is
+ // finished by the next pass, whatever stopped this one.
+ for _, s := range finished {
+ if err := os.Remove(s.markerPath); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ res.Errors = append(res.Errors, fmt.Errorf("%s: %w", s.id, err))
+ continue
+ }
+ delete(r.last, s.id)
+ delete(r.said, s.id)
+ res.Removed = append(res.Removed, s.id)
+ if err := r.hook(Point{Step: 7, Where: WhereDone, Session: s.id}); err != nil {
+ res.Errors = append(res.Errors, err)
+ return res
+ }
+ }
+ return res
+}
+
+func (r *Remover) now() time.Time {
+ if r.Now != nil {
+ return r.Now()
+ }
+ return time.Now()
+}
+
+func (r *Remover) hook(p Point) error {
+ if r.Hook == nil {
+ return nil
+ }
+ if err := r.Hook(p); err != nil {
+ return crash{err}
+ }
+ return nil
+}
+
+// keep reports a blocked reason once per process.
+func (r *Remover) keep(res *Result, session, reason string) {
+ if r.said[session] == nil {
+ r.said[session] = map[string]bool{}
+ }
+ if r.said[session][reason] {
+ return
+ }
+ r.said[session][reason] = true
+ res.Kept = append(res.Kept, Kept{Session: session, Reason: reason})
+}
+
+// markerNames lists the marker files, in name order.
+func (r *Remover) markerNames() ([]string, error) {
+ items, err := os.ReadDir(filepath.Join(r.Source, scenario.MarkerDir))
+ if err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil, nil
+ }
+ return nil, err
+ }
+ var out []string
+ for _, it := range items {
+ if it.Type().IsRegular() && scenario.IsMarkerName(it.Name()) {
+ out = append(out, it.Name())
+ }
+ }
+ return out, nil
+}
+
+type candidate struct {
+ id string
+ path string
+ m *scenario.Marker
+ err error
+}
+
+// candidates reads every marker. A removal that has begun goes first, so a
+// half-removed session is never left behind others that are only waiting.
+func (r *Remover) candidates(names []string) []candidate {
+ var removing, rest []candidate
+ for _, name := range names {
+ c := candidate{id: strings.TrimSuffix(name, ".json"), path: filepath.Join(r.Source, scenario.MarkerDir, name)}
+ c.m, c.err = scenario.ReadMarker(c.path)
+ if c.err == nil && c.m.State == scenario.MarkerRemoving {
+ removing = append(removing, c)
+ continue
+ }
+ rest = append(rest, c)
+ }
+ return append(removing, rest...)
+}
+
+// Blocked reports why no marked session can be removed with this input, or
+// an empty string when nothing at the level of the pass prevents it. It
+// checks what holds for every session or for none: that the pipeline sends
+// everything a session produces, that the changes adapter lands the plugin
+// output the build wrote, and that neither source directory is Claude
+// Code's own. It reads no marker and nothing in the storage root. So the
+// pipeline asks it before the first pass, and prints the answer, which then
+// agrees with what every pass does.
+func (r *Remover) Blocked(in Input) string {
+ if in.Endpoint == "" {
+ return "no export endpoint; nothing is sent, so nothing is removed"
+ }
+ if !in.SendLogs {
+ return "export.otlp.logs is off; landed files and rounds are never sent"
+ }
+ if !in.SendMetrics {
+ return "export.otlp.metrics is off; the metrics derived from a session are never sent"
+ }
+ want := resolved(filepath.Join(r.Source, "plugins", "data"))
+ if r.Changes == nil {
+ return fmt.Sprintf("the changes adapter is off; the plugin output the build wrote under %s would never be landed", want)
+ }
+ if got := resolved(r.ChangesRoot); got != want {
+ return fmt.Sprintf("the changes adapter reads %s, not %s; the plugin output the build wrote would never be landed", r.ChangesRoot, want)
+ }
+ // A source directory that is one of Claude Code's own, holds one, or
+ // lies inside one would let a marker path reach a real transcript. Each
+ // source is compared with every place Claude Code may keep its files,
+ // not only the one this process's environment selects. The pipeline can
+ // run with CLAUDE_CONFIG_DIR or XDG_CONFIG_HOME set differently from the
+ // Claude Code a person runs on the same machine.
+ for _, src := range []string{r.Source, r.ChangesRoot} {
+ for _, own := range claudeCodeDirs() {
+ if overlaps(src, own) {
+ return fmt.Sprintf("%s is, holds, or lies inside %s, where Claude Code keeps its own files; asz never removes there", src, own)
+ }
+ }
+ }
+ return ""
+}
+
+// claudeCodeDirs lists every directory where Claude Code may keep its
+// files. These are the three places claudecode.ResolveSourceRoot chooses
+// between: what CLAUDE_CONFIG_DIR names, XDG_CONFIG_HOME/claude, and
+// ~/.claude. Each is listed with its projects and plugins/data, because a
+// link can put either of them somewhere else, and each is resolved on its
+// own. The directory itself is listed too, so a source anywhere inside it is
+// refused. With no home directory and neither variable set, the list is
+// empty: the machine has no such directory to protect.
+func claudeCodeDirs() []string {
+ var bases []string
+ if d := os.Getenv("CLAUDE_CONFIG_DIR"); d != "" {
+ bases = append(bases, d)
+ }
+ if d := os.Getenv("XDG_CONFIG_HOME"); d != "" {
+ bases = append(bases, filepath.Join(d, "claude"))
+ }
+ if home, err := os.UserHomeDir(); err == nil && home != "" {
+ bases = append(bases, filepath.Join(home, ".claude"))
+ }
+ dirs := make([]string, 0, 3*len(bases))
+ for _, b := range bases {
+ dirs = append(dirs, b, filepath.Join(b, "projects"), filepath.Join(b, "plugins", "data"))
+ }
+ return dirs
+}
+
+// overlaps reports whether a and b are the same directory, or one lies
+// inside the other. They are compared as written, made absolute, and again
+// with symbolic links resolved, so a link on either side cannot hide it.
+func overlaps(a, b string) bool {
+ la, lb := absolute(a), absolute(b)
+ ra, rb := resolved(a), resolved(b)
+ return under(la, lb) || under(lb, la) || under(ra, rb) || under(rb, ra)
+}
+
+// pass is what one removal pass reads once and shares between sessions.
+type pass struct {
+ r *Remover
+ in Input
+ root string
+ now time.Time
+ sent *otlp.Sent
+ derived *metrics.Derived
+ // spool lists the spool files by the session that owns them.
+ spool map[string][]string
+
+ discovered bool
+ discovery verdict
+ local map[string]claudecode.Session
+ changes map[string]claudecodechanges.Session
+}
+
+// session is one marked session while a pass works on it.
+type session struct {
+ id string
+ markerPath string
+ m *scenario.Marker
+ sessionLock *storage.SessionLock
+ chainLock *storage.SessionLock
+ local claudecode.Session
+ hasLocal bool
+ changes claudecodechanges.Session
+ hasChanges bool
+}
+
+func (s *session) unlock() {
+ if s.chainLock != nil {
+ _ = s.chainLock.Unlock()
+ s.chainLock = nil
+ }
+ if s.sessionLock != nil {
+ _ = s.sessionLock.Unlock()
+ s.sessionLock = nil
+ }
+}
+
+// decide checks the conditions of one session in order, and holds its locks
+// when every one of them holds.
+func (p *pass) decide(s *session) verdict {
+ policy, _ := s.m.Removal()
+ if !policy.Immediate() {
+ if t, ok := p.r.last[s.id]; ok && p.now.Before(t.Add(policy.Retain)) {
+ return waiting("kept until %s", t.Add(policy.Retain).UTC().Format(time.RFC3339))
+ }
+ }
+ if p.in.Busy[s.id] {
+ return waiting("busy")
+ }
+ if v := p.lock(s); v.reason != "" {
+ return v
+ }
+ if v := p.whole(s); v.reason != "" {
+ return v
+ }
+ if v := p.covered(s); v.reason != "" {
+ return v
+ }
+ landed, v := p.landed(s)
+ if v.reason != "" {
+ return v
+ }
+ if v := p.sentAll(landed.files, "not sent yet"); v.reason != "" {
+ return v
+ }
+ if v := p.chainSent(s, landed); v.reason != "" {
+ return v
+ }
+ if v := p.metricsSent(s, landed); v.reason != "" {
+ return v
+ }
+ if !policy.Immediate() {
+ if !landed.timed {
+ return blocked("no landed record carries a time, so the retention cannot be counted")
+ }
+ if until := landed.last.Add(policy.Retain); p.now.Before(until) {
+ return waiting("kept until %s", until.UTC().Format(time.RFC3339))
+ }
+ }
+ return verdict{}
+}
+
+// lock takes the session's lock and its chain's lock. Each is taken only when
+// its directory exists, because taking a lock creates its directory.
+func (p *pass) lock(s *session) verdict {
+ take := func(dir string, lockFn func(string) (*storage.SessionLock, error), busy error) (*storage.SessionLock, verdict) {
+ fi, err := os.Lstat(dir)
+ switch {
+ case errors.Is(err, fs.ErrNotExist):
+ return nil, verdict{}
+ case err != nil:
+ return nil, blocked("%v", err)
+ case !fi.IsDir():
+ return nil, blocked("%s is a symbolic link or not a directory", dir)
+ }
+ l, err := lockFn(dir)
+ if errors.Is(err, busy) {
+ return nil, waiting("busy")
+ }
+ if err != nil {
+ return nil, blocked("%v", err)
+ }
+ return l, verdict{}
+ }
+ var v verdict
+ if s.sessionLock, v = take(p.r.Zone.SessionDir(s.id), storage.LockSession, storage.ErrSessionBusy); v.reason != "" {
+ return v
+ }
+ s.chainLock, v = take(filepath.Join(p.root, conversationsDir, s.id), storage.LockChain, storage.ErrChainBusy)
+ return v
+}
+
+// discover runs both adapters' discovery once per pass, and only when a
+// session gets this far. A pass where every session is waiting on its
+// retention discovers nothing.
+func (p *pass) discover() verdict {
+ if p.discovered {
+ return p.discovery
+ }
+ p.discovered = true
+ all := func(format string, a ...any) verdict {
+ v := blocked(format, a...)
+ v.every = true
+ return v
+ }
+ sessions, warnings, err := claudecode.DiscoverWithWarnings(p.r.Source)
+ if err == nil && len(warnings) > 0 {
+ err = warnings[0]
+ }
+ if err != nil {
+ p.discovery = all("discovery could not read everything under %s (%v); files of a session may be hidden", p.r.Source, err)
+ return p.discovery
+ }
+ p.local = map[string]claudecode.Session{}
+ for _, s := range sessions {
+ p.local[s.ID] = s
+ }
+ changed, warnings, err := claudecodechanges.DiscoverWithWarnings(p.r.ChangesRoot)
+ if err == nil && len(warnings) > 0 {
+ err = warnings[0]
+ }
+ if err != nil {
+ p.discovery = all("discovery could not read everything under %s (%v); files of a session may be hidden", p.r.ChangesRoot, err)
+ return p.discovery
+ }
+ p.changes = map[string]claudecodechanges.Session{}
+ for _, s := range changed {
+ p.changes[s.ID] = s
+ }
+ return p.discovery
+}
+
+// whole checks that the build finished the session and nothing changed it
+// since: the adapters find exactly the files the marker lists, each with the
+// bytes the build wrote, and their filters take the session.
+func (p *pass) whole(s *session) verdict {
+ if v := p.discover(); v.reason != "" {
+ return v
+ }
+ want := map[string]bool{}
+ for _, f := range s.m.Files {
+ want[f.Path] = true
+ }
+ found := map[string]bool{}
+ s.local, s.hasLocal = p.local[s.id]
+ for _, src := range s.local.Sources {
+ found[src.Rel] = true
+ }
+ s.changes, s.hasChanges = p.changes[s.id]
+ for _, src := range s.changes.Sources {
+ found[changesPrefix+src.Rel] = true
+ }
+ for _, rel := range sortedKeys(found) {
+ if !want[rel] {
+ return blocked("%s is not a file the build wrote", p.source(rel))
+ }
+ }
+ for _, f := range s.m.Files {
+ abs := p.source(f.Path)
+ fi, err := os.Lstat(abs)
+ switch {
+ case errors.Is(err, fs.ErrNotExist):
+ return blocked("%s, which the build wrote, is missing", abs)
+ case err != nil:
+ return blocked("%v", err)
+ case !fi.Mode().IsRegular():
+ return blocked("%s is a symbolic link or not a regular file", abs)
+ case !found[f.Path]:
+ return blocked("%s, which the build wrote, is not found by either adapter's discovery", abs)
+ case fi.Size() != f.Size:
+ return blocked("%s changed after the build wrote it", abs)
+ }
+ sum, err := storage.FileDigest(abs)
+ if err != nil {
+ return blocked("%v", err)
+ }
+ if sum != f.SHA256 {
+ return blocked("%s changed after the build wrote it", abs)
+ }
+ }
+ if s.hasLocal && p.r.LocalMatch != nil && !p.r.LocalMatch(s.local) {
+ return blocked("the local adapter's include and exclude leave %s out", s.id)
+ }
+ if s.hasChanges && p.r.ChangesMatch != nil && !p.r.ChangesMatch(s.changes) {
+ return blocked("the changes adapter's include and exclude leave %s out", s.id)
+ }
+ return verdict{}
+}
+
+// covered checks that each adapter would land nothing of the session now.
+// A source deleted with a line not yet landed would lose that line.
+func (p *pass) covered(s *session) verdict {
+ if s.hasLocal {
+ ok, reason, err := p.r.Local.Covered(s.local)
+ if err != nil {
+ return blocked("%v", err)
+ }
+ if !ok {
+ return waiting("%s", reason)
+ }
+ }
+ if s.hasChanges {
+ ok, reason, err := p.r.Changes.Covered(s.changes)
+ if err != nil {
+ return blocked("%v", err)
+ }
+ if !ok {
+ return waiting("%s", reason)
+ }
+ }
+ return verdict{}
+}
+
+// sentFile is a file that must be recorded as sent: its path under the root
+// with forward slashes, and its digest now.
+type sentFile struct{ rel, digest string }
+
+// landedSet is what a session's landed files hold.
+type landedSet struct {
+ files []sentFile
+ maxSeq uint64
+ // last is the latest record time, when timed.
+ last time.Time
+ timed bool
+}
+
+// landed reads every landed file of the session once: its header, its digest
+// and its latest record time. Every one must have come from a file the build
+// wrote. That keeps a real session whose id a scenario reused, and whose
+// transcript was copied beside the scenario's, from ever being removed.
+func (p *pass) landed(s *session) (*landedSet, verdict) {
+ files, err := storage.LandedFiles(p.r.Zone, s.id)
+ if err != nil {
+ return nil, blocked("%v", err)
+ }
+ if len(files) == 0 {
+ return nil, blocked("nothing of it is landed, so nothing of it can be sent")
+ }
+ want := map[string]bool{}
+ for _, f := range s.m.Files {
+ want[f.Path] = true
+ }
+ set := &landedSet{}
+ for _, lf := range files {
+ rel := p.rel(lf.Path)
+ info, err := readLanded(lf.Path)
+ if err != nil {
+ return nil, blocked("landed file %s cannot be read: %v", rel, err)
+ }
+ // The header names the adapter and its version, as name/version.
+ src := info.Src
+ switch name, _, _ := strings.Cut(info.Adapter, "/"); name {
+ case claudecode.Name:
+ case claudecodechanges.Name:
+ src = changesPrefix + src
+ default:
+ return nil, blocked("landed file %s came through the adapter %q, which reads no file a build writes", rel, info.Adapter)
+ }
+ if info.Session != s.id || !want[src] {
+ return nil, blocked("landed file %s names a source the build did not write", rel)
+ }
+ set.files = append(set.files, sentFile{rel: rel, digest: info.digest})
+ set.maxSeq = max(set.maxSeq, lf.Seq)
+ if info.timed && (!set.timed || info.last.After(set.last)) {
+ set.last, set.timed = info.last, true
+ }
+ }
+ if set.timed {
+ p.r.last[s.id] = set.last
+ }
+ return set, verdict{}
+}
+
+// sentAll checks that every file was sent to this pipeline's endpoint with
+// the digest it has now, and that no receiver rejected records of it. A
+// blocked reason wins over a waiting one, since it will not change on its
+// own.
+func (p *pass) sentAll(files []sentFile, notYet string) verdict {
+ var wait verdict
+ for _, f := range files {
+ d, ok := p.sent.Digest(f.rel)
+ switch {
+ case !ok:
+ if wait.reason == "" {
+ wait = waiting("%s", notYet)
+ }
+ case p.sent.Rejected(f.rel):
+ return blocked("the receiver rejected records of %s", f.rel)
+ case d != f.digest:
+ return blocked("%s changed after it was sent", f.rel)
+ case !p.sent.SentTo(p.in.Endpoint, f.rel, f.digest):
+ return blocked("%s is not recorded as sent to %s", f.rel, p.in.Endpoint)
+ }
+ }
+ return wait
+}
+
+// roundName is the name a chain gives a round, as the pusher reads it.
+var roundName = regexp.MustCompile(`^r(\d{6,})-([0-9a-f]{12})\.sf$`)
+
+// chainSent checks that the chain reaches the last landed file, and that
+// every round of it is sent. A round still being written is never pushed, so
+// it never passes.
+func (p *pass) chainSent(s *session, landed *landedSet) verdict {
+ const notYet = "not parsed or not sent yet"
+ dir := filepath.Join(p.root, conversationsDir, s.id, "rounds")
+ items, err := os.ReadDir(dir)
+ if errors.Is(err, fs.ErrNotExist) {
+ return waiting(notYet)
+ }
+ if err != nil {
+ return blocked("%v", err)
+ }
+ var rounds []sentFile
+ var head string
+ var headN uint64
+ for _, it := range items {
+ if it.IsDir() || !strings.HasSuffix(it.Name(), ".sf") {
+ continue
+ }
+ path := filepath.Join(dir, it.Name())
+ m := roundName.FindStringSubmatch(it.Name())
+ if m == nil {
+ return blocked("%s has a name no chain writes", p.rel(path))
+ }
+ n, _ := strconv.ParseUint(m[1], 10, 64)
+ if head == "" || n > headN {
+ head, headN = path, n
+ }
+ digest, err := storage.FileDigest(path)
+ if err != nil {
+ return blocked("%v", err)
+ }
+ rounds = append(rounds, sentFile{rel: p.rel(path), digest: digest})
+ }
+ if head == "" {
+ return waiting(notYet)
+ }
+ hdr, err := readRoundHeader(head)
+ if err != nil {
+ return blocked("the head round %s cannot be read: %v", p.rel(head), err)
+ }
+ if hdr.Conversation != s.id || hdr.Session != s.id {
+ return blocked("the head round %s names conversation %q and session %q, not %s", p.rel(head), hdr.Conversation, hdr.Session, s.id)
+ }
+ switch {
+ case hdr.ThroughSeq < landed.maxSeq:
+ return waiting(notYet)
+ case hdr.ThroughSeq > landed.maxSeq:
+ return blocked("the head round %s reaches sequence %d, past the last landed file, %d", p.rel(head), hdr.ThroughSeq, landed.maxSeq)
+ }
+ return p.sentAll(rounds, notYet)
+}
+
+// metricsSent checks that metrics were derived from every landed file, when
+// the pipeline derives them, and that every spool file of the session is
+// sent. A spool file of an earlier configuration counts either way.
+func (p *pass) metricsSent(s *session, landed *landedSet) verdict {
+ if p.r.Derives {
+ for _, f := range landed.files {
+ if !p.derived.Has(f.rel, f.digest) {
+ v := waiting("metrics not derived yet")
+ v.derive = true
+ return v
+ }
+ }
+ }
+ var spool []sentFile
+ for _, name := range p.spool[s.id] {
+ path := filepath.Join(p.root, storage.SpoolDir, name)
+ digest, err := storage.FileDigest(path)
+ if err != nil {
+ return blocked("%v", err)
+ }
+ spool = append(spool, sentFile{rel: storage.SpoolDir + "/" + name, digest: digest})
+ }
+ return p.sentAll(spool, "metrics not sent yet")
+}
+
+// source is a marker path as a path on disk.
+func (p *pass) source(rel string) string {
+ return filepath.Join(p.r.Source, filepath.FromSlash(rel))
+}
+
+// rel is a path under the storage root as push.state and metrics.state
+// write it.
+func (p *pass) rel(path string) string {
+ rel, err := filepath.Rel(p.root, path)
+ if err != nil {
+ return filepath.ToSlash(path)
+ }
+ return filepath.ToSlash(rel)
+}
+
+// run takes a session through steps 0 to 4. The order is what makes each
+// step safe to stop at and to run again:
+//
+// - The source goes before any cursor. The cursors live in <root>/S, and a
+// source that outlived them would be read again from byte 0, landed under
+// new names and sent again.
+// - The landed files go before the chain. With them gone, nothing parses S
+// again, and every round left is already sent. The other way round, a
+// restart would parse the whole landed range into a new round 1, whose
+// name carries a new digest, and send it again.
+// - Each directory goes in one rename. Deleting one in place takes a call
+// per file, and a crash in the middle left a session with part of its
+// landed files and no index. A parse then published a round of
+// tombstones over what was left.
+func (r *Remover) run(s *session) error {
+ if s.m.State != scenario.MarkerRemoving {
+ m := *s.m
+ m.State = scenario.MarkerRemoving
+ if err := scenario.WriteMarker(s.markerPath, &m); err != nil {
+ return err
+ }
+ s.m = &m
+ if err := r.hook(Point{Step: 0, Where: WhereDone, Session: s.id}); err != nil {
+ return err
+ }
+ }
+ if err := r.removeSource(s); err != nil {
+ return err
+ }
+ root := r.Zone.Root()
+ if err := r.removeTree(s, 2, r.Zone.SessionDir(s.id), s.id, &s.sessionLock); err != nil {
+ return err
+ }
+ // The name is outside _conversations. The pusher and the page list every
+ // directory there, and a round under a new path would be sent again.
+ if err := r.removeTree(s, 3, filepath.Join(root, conversationsDir, s.id), s.id+".chain", &s.chainLock); err != nil {
+ return err
+ }
+ return r.removeSpool(s)
+}
+
+// removeSource is step 1: every file the marker lists, then the directories
+// named for the session. A file already gone was deleted by a removal that
+// stopped part way, so it counts as done.
+func (r *Remover) removeSource(s *session) error {
+ source := resolved(r.Source)
+ for k, f := range s.m.Files {
+ abs := filepath.Join(r.Source, filepath.FromSlash(f.Path))
+ fi, err := os.Lstat(abs)
+ switch {
+ case errors.Is(err, fs.ErrNotExist):
+ case err != nil:
+ return err
+ case !fi.Mode().IsRegular():
+ return stop{abs, "is a symbolic link or not a regular file"}
+ default:
+ if !under(resolved(filepath.Dir(abs)), source) {
+ return stop{abs, "is reached through a symbolic link that leads out of " + r.Source}
+ }
+ if fi.Size() != f.Size {
+ return stop{abs, "changed after the build wrote it"}
+ }
+ sum, err := storage.FileDigest(abs)
+ if err != nil {
+ return err
+ }
+ if sum != f.SHA256 {
+ return stop{abs, "changed after the build wrote it"}
+ }
+ if err := os.Remove(abs); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ }
+ if err := r.hook(Point{Step: 1, Where: WhereFile, Session: s.id, N: k + 1}); err != nil {
+ return err
+ }
+ }
+ // Only directories named for the session, and only when empty. A project
+ // directory, which every session of a feed shares, is never touched, and
+ // neither is the plugin's directory nor the marker directory.
+ for _, dir := range dirsNamedFor(s.id, s.m.Files) {
+ abs := filepath.Join(r.Source, filepath.FromSlash(dir))
+ fi, err := os.Lstat(abs)
+ if err != nil || !fi.IsDir() {
+ continue
+ }
+ if !under(resolved(abs), source) {
+ return stop{abs, "is reached through a symbolic link that leads out of " + r.Source}
+ }
+ items, err := os.ReadDir(abs)
+ if err != nil {
+ return err
+ }
+ if len(items) > 0 {
+ continue
+ }
+ if err := os.Remove(abs); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ }
+ return r.hook(Point{Step: 1, Where: WhereDirs, Session: s.id})
+}
+
+// removeTree is steps 2 and 3: one directory, renamed away whole and then
+// deleted. The lock inside it is released first, on every platform, since
+// Windows cannot rename a directory that holds a file opened with no
+// sharing, which is how a lock is held there.
+func (r *Remover) removeTree(s *session, step int, dir, name string, held **storage.SessionLock) error {
+ fi, err := os.Lstat(dir)
+ switch {
+ case err == nil && !fi.IsDir():
+ return stop{dir, "is a symbolic link or not a directory"}
+ case err != nil && !errors.Is(err, fs.ErrNotExist):
+ return err
+ }
+ if *held != nil {
+ _ = (*held).Unlock()
+ *held = nil
+ }
+ if err := r.hook(Point{Step: step, Where: WhereUnlocked, Session: s.id}); err != nil {
+ return err
+ }
+ err = storage.RemoveDir(r.Zone.Root(), dir, name, func(where string) error {
+ return r.hook(Point{Step: step, Where: where, Session: s.id})
+ })
+ if err != nil {
+ var c crash
+ if !errors.As(err, &c) && errors.Is(err, storage.ErrExists) {
+ return stop{filepath.Join(r.Zone.Root(), storage.RemovedDir, name), "is left from an earlier removal and could not be deleted"}
+ }
+ return err
+ }
+ return r.hook(Point{Step: step, Where: WhereDone, Session: s.id})
+}
+
+// removeSpool is step 4: every spool file the session owns.
+func (r *Remover) removeSpool(s *session) error {
+ dir := filepath.Join(r.Zone.Root(), storage.SpoolDir)
+ items, err := os.ReadDir(dir)
+ if err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ k := 0
+ for _, it := range items {
+ if it.IsDir() {
+ continue
+ }
+ if owner, ok := metrics.SpoolOwner(it.Name()); !ok || owner != s.id {
+ continue
+ }
+ // A spool file is read-only, as a landed file is. DeleteTree makes it
+ // writable before it deletes it.
+ if err := storage.DeleteTree(filepath.Join(dir, it.Name()), nil); err != nil {
+ return err
+ }
+ k++
+ if err := r.hook(Point{Step: 4, Where: WhereFile, Session: s.id, N: k}); err != nil {
+ return err
+ }
+ }
+ return r.hook(Point{Step: 4, Where: WhereDone, Session: s.id})
+}
+
+// dirsNamedFor lists, deepest first, the directories of the marker's files
+// that sit at or below a path segment equal to the session id.
+func dirsNamedFor(session string, files []scenario.MarkerFile) []string {
+ seen := map[string]bool{}
+ for _, f := range files {
+ parts := strings.Split(f.Path, "/")
+ for i := 0; i < len(parts)-1; i++ {
+ if parts[i] != session {
+ continue
+ }
+ for j := i + 1; j < len(parts); j++ {
+ seen[strings.Join(parts[:j], "/")] = true
+ }
+ break
+ }
+ }
+ out := sortedKeys(seen)
+ sort.SliceStable(out, func(i, j int) bool {
+ di, dj := strings.Count(out[i], "/"), strings.Count(out[j], "/")
+ if di != dj {
+ return di > dj
+ }
+ return out[i] > out[j]
+ })
+ return out
+}
+
+// ownedBy says which lines of push.state belong to the sessions given:
+// their landed files, their rounds and the spool files they own.
+func ownedBy(ids map[string]bool) func(rel string) bool {
+ return func(rel string) bool {
+ first, rest, ok := strings.Cut(rel, "/")
+ if !ok || rest == "" {
+ return false
+ }
+ switch first {
+ case conversationsDir:
+ conv, _, _ := strings.Cut(rest, "/")
+ return ids[conv]
+ case storage.SpoolDir:
+ owner, ok := metrics.SpoolOwner(rest)
+ return ok && !strings.Contains(rest, "/") && ids[owner]
+ }
+ return ids[first]
+ }
+}
+
+// spoolOwners lists the spool files by the session that owns them. A
+// request the receiver adapter put there names no session and is left out.
+func spoolOwners(root string) (map[string][]string, error) {
+ items, err := os.ReadDir(filepath.Join(root, storage.SpoolDir))
+ if err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ return map[string][]string{}, nil
+ }
+ return nil, err
+ }
+ out := map[string][]string{}
+ for _, it := range items {
+ if it.IsDir() {
+ continue
+ }
+ if owner, ok := metrics.SpoolOwner(it.Name()); ok {
+ out[owner] = append(out[owner], it.Name())
+ }
+ }
+ return out, nil
+}
+
+// landedInfo is what a removal reads from a landed file.
+type landedInfo struct {
+ Adapter string `json:"adapter"`
+ Src string `json:"src"`
+ Session string `json:"session"`
+ digest string
+ last time.Time
+ timed bool
+}
+
+// readLanded reads a landed file once: the header on its first line, the
+// SHA-256 of all of it, and the latest record time, by the rule the pusher
+// uses to stamp a file. Lines reach a megabyte and more, so a line is read
+// to its newline rather than with a scanner.
+func readLanded(path string) (landedInfo, error) {
+ var info landedInfo
+ f, err := os.Open(path)
+ if err != nil {
+ return info, err
+ }
+ defer f.Close()
+ h := sha256.New()
+ br := bufio.NewReaderSize(io.TeeReader(f, h), 1<<20)
+ first := true
+ var hi int64
+ for {
+ line, err := br.ReadBytes('\n')
+ if len(line) > 0 {
+ line = bytes.TrimRight(line, "\n")
+ if first {
+ if jerr := json.Unmarshal(line, &info); jerr != nil {
+ return info, fmt.Errorf("the header: %w", jerr)
+ }
+ first = false
+ } else if ns, ok := sessiondata.LineTime(line); ok && (!info.timed || ns > hi) {
+ hi, info.timed = ns, true
+ }
+ }
+ if errors.Is(err, io.EOF) {
+ break
+ }
+ if err != nil {
+ return info, err
+ }
+ }
+ if first {
+ return info, errors.New("it has no header")
+ }
+ info.digest = hex.EncodeToString(h.Sum(nil))
+ if info.timed {
+ info.last = time.Unix(0, hi).UTC()
+ }
+ return info, nil
+}
+
+// roundHeader is the part of a round's first line a removal reads. The field
+// names are Session Flow's, written out because its package is server side.
+type roundHeader struct {
+ Conversation string `json:"conversation"`
+ Session string `json:"session"`
+ ThroughSeq uint64 `json:"through_seq"`
+}
+
+func readRoundHeader(path string) (roundHeader, error) {
+ var hdr roundHeader
+ f, err := os.Open(path)
+ if err != nil {
+ return hdr, err
+ }
+ defer f.Close()
+ line, err := bufio.NewReader(f).ReadBytes('\n')
+ if err != nil && !errors.Is(err, io.EOF) {
+ return hdr, err
+ }
+ return hdr, json.Unmarshal(bytes.TrimRight(line, "\n"), &hdr)
+}
+
+// absolute is a path made absolute and clean, with nothing resolved.
+func absolute(p string) string {
+ if abs, err := filepath.Abs(p); err == nil {
+ return abs
+ }
+ return filepath.Clean(p)
+}
+
+// resolved is a path made absolute and clean, with symbolic links resolved
+// as far as the path exists.
+func resolved(p string) string {
+ abs := absolute(p)
+ if target, err := filepath.EvalSymlinks(abs); err == nil {
+ return target
+ }
+ parent := filepath.Dir(abs)
+ if parent == abs {
+ return abs
+ }
+ return filepath.Join(resolved(parent), filepath.Base(abs))
+}
+
+// under reports whether p is dir or inside it. Both are absolute and clean.
+func under(p, dir string) bool {
+ rel, err := filepath.Rel(dir, p)
+ return err == nil && (rel == "." || filepath.IsLocal(rel))
+}
+
+func sortedKeys(m map[string]bool) []string {
+ out := make([]string, 0, len(m))
+ for k := range m {
+ out = append(out, k)
+ }
+ sort.Strings(out)
+ return out
+}
diff --git a/internal/scenario/remove/remove_test.go b/internal/scenario/remove/remove_test.go
new file mode 100644
index 0000000..0b53da8
--- /dev/null
+++ b/internal/scenario/remove/remove_test.go
@@ -0,0 +1,861 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package remove_test
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecodechanges"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp/otlptest"
+ "github.com/apache/skywalking-ai-sessionizer/internal/metrics"
+ "github.com/apache/skywalking-ai-sessionizer/internal/parse"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario/remove"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// everyKind is one session holding every kind of file a removal deletes: a
+// main transcript, a child with a meta file, a workflow whose script is
+// filed under another project, and plugin output.
+const everyKind = `title: every kind of file a removal deletes
+steps:
+ - input: fix the timeout and check everything
+ - call:
+ text: Fixing it.
+ tool:
+ name: Bash
+ input: {command: "sed -i 's/= 10/= 30/' server.go"}
+ result: {text: "", after: 500ms}
+ changes:
+ - {path: server.go, before: "package main\n\nvar timeout = 10\n", after: "package main\n\nvar timeout = 30\n"}
+ usage: {in: 2, out: 50, cache_read: 900, cache_write: 100}
+ - call:
+ agent: {name: helper, prompt: look at the tests, steps: [{call: {text: tests look fine, usage: {out: 20}}}], notify: true}
+ - call:
+ workflow:
+ name: verify
+ script_project: -Users-dev-script-filed-elsewhere
+ children:
+ - {name: lint, prompt: run the linter, steps: [{call: {text: lint done}}]}
+ - call: {text: Everything checks out.}
+`
+
+var at = time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+
+// root is a scenario root and the receiver its pipeline sends to.
+type root struct {
+ t *testing.T
+ dir string
+ source string
+ zone *storage.Zone
+ rcv *otlptest.Receiver
+ endpoint string
+}
+
+func newRoot(t *testing.T) *root {
+ t.Helper()
+ // Claude Code's own directory is a place of the test's, so the guard
+ // on it compares against something known on every machine.
+ t.Setenv("CLAUDE_CONFIG_DIR", t.TempDir())
+ rcv, err := otlptest.Start()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(rcv.Close)
+ dir := t.TempDir()
+ o := rcv.Options(otlp.ProtocolGRPC)
+ return &root{t: t, dir: dir, source: filepath.Join(dir, "_source"), zone: storage.NewZone(dir), rcv: rcv,
+ endpoint: otlp.EndpointOf(o.Protocol, o.Endpoint, o.TLS)}
+}
+
+func load(t *testing.T, body string) *scenario.Scenario {
+ t.Helper()
+ path := filepath.Join(t.TempDir(), "scenario.yaml")
+ if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ sc, err := scenario.Load(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return sc
+}
+
+func (r *root) build(sc *scenario.Scenario, id string, opts scenario.Options) *scenario.Built {
+ r.t.Helper()
+ one := *sc
+ if id != "" {
+ one.Session = id
+ }
+ if opts.At.IsZero() {
+ opts.At = at
+ }
+ b, err := scenario.Build(&one, scenario.FormatClaudeCode, r.dir, opts)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ return b
+}
+
+func (r *root) changesRoot() string { return filepath.Join(r.source, "plugins", "data") }
+
+// collect lands with both adapters and returns how many sources landed.
+func (r *root) collect() int {
+ r.t.Helper()
+ st, err := claudecode.New(r.source, r.zone, 0).CollectAll(nil)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ cs, err := claudecodechanges.New(r.changesRoot(), r.zone, 0).CollectAll(nil)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ if errs := append(st.Errors, cs.Errors...); len(errs) > 0 {
+ r.t.Fatalf("collect: %v", errs)
+ }
+ return st.SourcesLanded + cs.SourcesLanded
+}
+
+// derive writes the metrics of what is landed and returns how many requests
+// it put in the spool.
+func (r *root) derive() int {
+ r.t.Helper()
+ st, err := (&metrics.Deriver{Zone: r.zone, Options: metrics.Options{Grace: -1, Version: "test"}}).Pass(nil)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ if len(st.Errors) > 0 {
+ r.t.Fatalf("derive: %v", st.Errors)
+ }
+ return st.Requests
+}
+
+// parse writes rounds for every session directory, as a full pass does, and
+// returns how many it wrote.
+func (r *root) parse() int {
+ r.t.Helper()
+ items, err := os.ReadDir(r.dir)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ written := 0
+ for _, it := range items {
+ if !it.IsDir() || strings.HasPrefix(it.Name(), "_") {
+ continue
+ }
+ for {
+ res, err := parse.Session(r.zone, parse.Options{Conversation: it.Name(), Session: it.Name(), MaxRoundBytes: 4096})
+ if err != nil {
+ r.t.Fatalf("parse %s: %v", it.Name(), err)
+ }
+ if res.Changed() {
+ written++
+ }
+ if !res.Changed() || !res.More {
+ break
+ }
+ }
+ }
+ return written
+}
+
+// push sends what is not sent yet, recording endpoint as the receiver.
+func (r *root) push(endpoint string) *otlp.Stats {
+ r.t.Helper()
+ client, err := r.rcv.Client(otlp.ProtocolGRPC)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ defer client.Close()
+ p := &otlp.Pusher{Zone: r.zone, Client: client, Endpoint: endpoint, Version: "test",
+ ServiceName: "Remove Test", InstanceID: "remove-test", Layer: "AI_AGENT", MetricsService: "Remove Test"}
+ if err := p.Prepare(); err != nil {
+ r.t.Fatal(err)
+ }
+ st, err := p.Pass()
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ if len(st.Errors) > 0 {
+ r.t.Fatalf("push: %v", st.Errors)
+ }
+ return st
+}
+
+// pipeline is one pass of asz collect before its removal.
+func (r *root) pipeline() {
+ r.t.Helper()
+ r.collect()
+ r.derive()
+ r.parse()
+ r.push(r.endpoint)
+}
+
+func (r *root) remover() *remove.Remover {
+ return &remove.Remover{Zone: r.zone, Source: r.source, Local: claudecode.New(r.source, r.zone, 0),
+ Changes: claudecodechanges.New(r.changesRoot(), r.zone, 0), ChangesRoot: r.changesRoot(), Derives: true}
+}
+
+func (r *root) input() remove.Input {
+ return remove.Input{Endpoint: r.endpoint, SendLogs: true, SendMetrics: true}
+}
+
+func marker(t *testing.T, b *scenario.Built) *scenario.Marker {
+ t.Helper()
+ m, err := scenario.ReadMarker(b.Marker)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return m
+}
+
+func exists(p string) bool {
+ _, err := os.Lstat(p)
+ return !errors.Is(err, fs.ErrNotExist)
+}
+
+func noErrors(t *testing.T, res *remove.Result) {
+ t.Helper()
+ if len(res.Errors) > 0 {
+ t.Fatalf("the removal reported %v", res.Errors)
+ }
+}
+
+// removed checks that nothing of the session is left, and that nothing
+// every session shares went with it.
+func (r *root) removed(b *scenario.Built, m *scenario.Marker) {
+ r.t.Helper()
+ t, s := r.t, b.Session
+ if exists(b.Marker) {
+ t.Errorf("the marker %s is still there", b.Marker)
+ }
+ for _, f := range m.Files {
+ if exists(filepath.Join(r.source, filepath.FromSlash(f.Path))) {
+ t.Errorf("the source file %s is still there", f.Path)
+ }
+ if i := strings.Index(f.Path, "/"+s+"/"); i >= 0 {
+ if dir := filepath.Join(r.source, filepath.FromSlash(f.Path[:i+1+len(s)])); exists(dir) {
+ t.Errorf("the directory %s, named for the session, is still there", dir)
+ }
+ }
+ }
+ for _, p := range []string{filepath.Join(r.dir, s), filepath.Join(r.dir, "_conversations", s)} {
+ if exists(p) {
+ t.Errorf("%s is still there", p)
+ }
+ }
+ if items, err := os.ReadDir(filepath.Join(r.dir, storage.SpoolDir)); err == nil {
+ for _, it := range items {
+ if owner, ok := metrics.SpoolOwner(it.Name()); ok && owner == s {
+ t.Errorf("the spool file %s is still there", it.Name())
+ }
+ }
+ }
+ for _, state := range []string{filepath.Join(r.dir, otlp.StateFile), filepath.Join(r.dir, storage.SpoolDir, metrics.StateFile)} {
+ if data, err := os.ReadFile(state); err == nil && bytes.Contains(data, []byte(s)) {
+ t.Errorf("%s still names the session", state)
+ }
+ }
+ if items, err := os.ReadDir(filepath.Join(r.dir, storage.RemovedDir)); err == nil && len(items) > 0 {
+ t.Errorf("%s still holds %d entries", storage.RemovedDir, len(items))
+ }
+ for _, p := range []string{
+ filepath.Join(r.source, "-Users-dev-scenario", "not-a-uuid.jsonl"),
+ filepath.Join(r.source, "-Users-dev-scenario", "memory", "notes.md"),
+ filepath.Join(r.source, scenario.MarkerDir),
+ filepath.Join(r.dir, storage.ScenarioDir),
+ filepath.Join(r.source, filepath.FromSlash(scenario.PluginOutputDir)),
+ } {
+ if !exists(p) {
+ t.Errorf("%s, which every session shares, was removed", p)
+ }
+ }
+}
+
+// TestASentSessionIsRemovedWhole. Once everything the session produced is
+// recorded as sent to the endpoint, one pass removes all of it, and what
+// comes after finds nothing to land, derive, parse or send.
+func TestASentSessionIsRemovedWhole(t *testing.T) {
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ m := marker(t, b)
+ r.pipeline()
+
+ rm := r.remover()
+ res := rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 1 || res.Removed[0] != b.Session {
+ t.Fatalf("removed %v, kept %v, waiting %v; want %s removed", res.Removed, res.Kept, res.Waiting, b.Session)
+ }
+ r.removed(b, m)
+ data, err := os.ReadFile(filepath.Join(r.dir, otlp.StateFile))
+ if err != nil || !bytes.Contains(data, []byte("endpoint "+r.endpoint)) {
+ t.Fatalf("push.state lost its endpoint line: %v\n%s", err, data)
+ }
+
+ // Afterwards the session is not made again by anything.
+ r.rcv.Reset()
+ if n := r.collect(); n != 0 {
+ t.Errorf("the adapters landed %d source(s) of a removed session", n)
+ }
+ if n := r.derive(); n != 0 {
+ t.Errorf("the deriver wrote %d request(s) after the removal", n)
+ }
+ if n := r.parse(); n != 0 {
+ t.Errorf("the parse wrote %d round(s) after the removal", n)
+ }
+ r.push(r.endpoint)
+ if n := len(r.rcv.Requests()) + len(r.rcv.MetricsRequests()); n != 0 {
+ t.Errorf("the push sent %d request(s) after the removal", n)
+ }
+ r.removed(b, m)
+ again := rm.Pass(r.input())
+ noErrors(t, again)
+ if len(again.Removed) != 0 {
+ t.Errorf("a second pass removed %v", again.Removed)
+ }
+}
+
+// TestNothingIsRemovedBeforeAllOfItIsSent. A session goes only once its
+// landed files, its rounds and its metrics are all recorded as sent. Until
+// then it waits, and nothing is said about it, since it changes on its own.
+func TestNothingIsRemovedBeforeAllOfItIsSent(t *testing.T) {
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ r.collect()
+ r.parse()
+ rm := r.remover()
+
+ res := rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || len(res.Kept) != 0 || !waitingFor(res, "", "not sent yet") {
+ t.Fatalf("before any push: removed %v, kept %v, waiting %v", res.Removed, res.Kept, res.Waiting)
+ }
+ if marker(t, b).State != scenario.MarkerWritten {
+ t.Fatal("a pass that removed nothing changed the marker")
+ }
+
+ // Sent, but no metrics were derived yet. The session is carried into
+ // the next derivation, since it no longer lands anything.
+ r.push(r.endpoint)
+ res = rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !waitingFor(res, b.Session, "metrics not derived yet") {
+ t.Fatalf("with no metrics derived: removed %v, waiting %v", res.Removed, res.Waiting)
+ }
+ if len(res.DeriveNext) != 1 || res.DeriveNext[0] != b.Session {
+ t.Fatalf("DeriveNext is %v, want the session", res.DeriveNext)
+ }
+
+ r.derive()
+ res = rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !waitingFor(res, b.Session, "metrics not sent yet") {
+ t.Fatalf("with metrics derived and not sent: removed %v, waiting %v", res.Removed, res.Waiting)
+ }
+
+ r.push(r.endpoint)
+ res = rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 1 {
+ t.Fatalf("once all of it was sent: removed %v, kept %v, waiting %v", res.Removed, res.Kept, res.Waiting)
+ }
+}
+
+func waitingFor(res *remove.Result, session, reason string) bool {
+ for _, w := range res.Waiting {
+ if w.Session == session && strings.Contains(w.Reason, reason) {
+ return true
+ }
+ }
+ return false
+}
+
+func keptFor(res *remove.Result, session, reason string) bool {
+ for _, k := range res.Kept {
+ if k.Session == session && strings.Contains(k.Reason, reason) {
+ return true
+ }
+ }
+ return false
+}
+
+// TestARejectionKeepsTheSession. A receiver that took a request and
+// rejected records in it holds less than was sent, so the session is not
+// sent whole and is kept. The record of it survives the saves later
+// requests make.
+func TestARejectionKeepsTheSession(t *testing.T) {
+ r := newRoot(t)
+ sc := load(t, everyKind)
+ b := r.build(sc, "", scenario.Options{})
+ r.rcv.Reject(1)
+ r.collect()
+ r.derive()
+ r.parse()
+ r.push(r.endpoint)
+ rm := r.remover()
+ res := rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !keptFor(res, b.Session, "rejected records") {
+ t.Fatalf("after a partial success: removed %v, kept %v", res.Removed, res.Kept)
+ }
+
+ // Another session, sent whole. Its push saves push.state again.
+ r.rcv.Reject(0)
+ other := r.build(sc, "aaaaaaaa-bbbb-4ccc-8ddd-000000000001", scenario.Options{At: at.Add(time.Hour)})
+ r.pipeline()
+ res = rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 1 || res.Removed[0] != other.Session {
+ t.Fatalf("removed %v, want only %s; kept %v", res.Removed, other.Session, res.Kept)
+ }
+ if !exists(filepath.Join(r.dir, b.Session)) || !exists(b.Marker) {
+ t.Fatal("the session whose records were rejected was removed after a later save of push.state")
+ }
+ if keptFor(res, b.Session, "rejected") {
+ t.Fatal("the same reason was reported twice by one process")
+ }
+}
+
+// TestAnotherEndpointKeepsEverySession. push.state records the receivers
+// files went to, not which file went to which. Sent somewhere else, nothing
+// here is proved sent to this pipeline's receiver.
+func TestAnotherEndpointKeepsEverySession(t *testing.T) {
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ r.pipeline()
+ in := r.input()
+ in.Endpoint = "grpc://elsewhere.example:11800"
+ res := r.remover().Pass(in)
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !keptFor(res, "", "push.state records sending to "+r.endpoint) {
+ t.Fatalf("removed %v, kept %v", res.Removed, res.Kept)
+ }
+ if !exists(b.Marker) {
+ t.Fatal("the marker went")
+ }
+}
+
+// TestASessionWithNoMarkerIsNeverRemoved. A real Claude Code session has no
+// marker. Sent or not, nothing of it is touched.
+func TestASessionWithNoMarkerIsNeverRemoved(t *testing.T) {
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ m := marker(t, b)
+ r.pipeline()
+ if err := os.Remove(b.Marker); err != nil {
+ t.Fatal(err)
+ }
+ res := r.remover().Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || len(res.Kept) != 0 {
+ t.Fatalf("removed %v, kept %v", res.Removed, res.Kept)
+ }
+ for _, f := range m.Files {
+ if !exists(filepath.Join(r.source, filepath.FromSlash(f.Path))) {
+ t.Errorf("%s was deleted", f.Path)
+ }
+ }
+ for _, p := range []string{filepath.Join(r.dir, b.Session), filepath.Join(r.dir, "_conversations", b.Session)} {
+ if !exists(p) {
+ t.Errorf("%s was deleted", p)
+ }
+ }
+}
+
+// TestRetentionCountsFromTheLastRecord. A retained session stays until its
+// last record is as old as the policy says, counted from the records and
+// never from when anything ran, so the same data decides the same way.
+func TestRetentionCountsFromTheLastRecord(t *testing.T) {
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{Remove: scenario.Removal{Retain: 24 * time.Hour}})
+ m := marker(t, b)
+ last := lastRecord(t, r.source, m)
+ r.pipeline()
+
+ now := last.Add(24*time.Hour - time.Second)
+ rm := r.remover()
+ rm.Now = func() time.Time { return now }
+ res := rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !waitingFor(res, b.Session, "kept until") {
+ t.Fatalf("a second before its retention ends: removed %v, waiting %v", res.Removed, res.Waiting)
+ }
+ // The same remover, so its memory of the last record is used as well.
+ now = last.Add(24 * time.Hour)
+ res = rm.Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 1 {
+ t.Fatalf("once its retention ended: removed %v, waiting %v, kept %v", res.Removed, res.Waiting, res.Kept)
+ }
+ r.removed(b, m)
+}
+
+// lastRecord is the latest time among the lines of the source files, read
+// without the landing code.
+func lastRecord(t *testing.T, source string, m *scenario.Marker) time.Time {
+ t.Helper()
+ var last time.Time
+ for _, f := range m.Files {
+ data, err := os.ReadFile(filepath.Join(source, filepath.FromSlash(f.Path)))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, line := range bytes.Split(data, []byte("\n")) {
+ var rec map[string]any
+ if json.Unmarshal(line, &rec) != nil {
+ continue
+ }
+ for _, k := range []string{"timestamp", "time"} {
+ if s, ok := rec[k].(string); ok {
+ if ts, err := time.Parse(time.RFC3339Nano, s); err == nil && ts.After(last) {
+ last = ts
+ }
+ }
+ }
+ }
+ }
+ if last.IsZero() {
+ t.Fatal("no source line carries a time")
+ }
+ return last
+}
+
+// TestNeverInClaudeCodesOwnDirectory. A source directory that is Claude
+// Code's own, lies inside it, or holds it, could lead a marker to a real
+// transcript. Nothing is removed there, whatever the markers say.
+//
+// Every place Claude Code may keep its files is guarded, not only the one
+// the pipeline's environment selects. The pipeline can run with
+// CLAUDE_CONFIG_DIR or XDG_CONFIG_HOME set differently from the Claude Code
+// a person runs, which keeps its files in ~/.claude when neither is set.
+func TestNeverInClaudeCodesOwnDirectory(t *testing.T) {
+ const id = "c30736f2-ac0c-4a72-89b1-01a0844af62d"
+ for _, c := range []struct {
+ name string
+ // claude and xdg are where CLAUDE_CONFIG_DIR and XDG_CONFIG_HOME
+ // point, under the test's directory. Empty leaves the variable
+ // unset. The home directory is always home, under the same.
+ claude, xdg string
+ // source is the source directory. With link set, source is a
+ // symbolic link to link.
+ source, link string
+ guarded bool
+ }{
+ {name: "equal", claude: "cfg", source: "cfg/projects", guarded: true},
+ {name: "inside", claude: "cfg", source: "cfg/projects/sub", guarded: true},
+ {name: "holding it", claude: "cfg", source: "cfg", guarded: true},
+ {name: "in ~/.claude with CLAUDE_CONFIG_DIR elsewhere", claude: "cfg", source: "home/.claude/projects", guarded: true},
+ {name: "in ~/.claude with XDG_CONFIG_HOME elsewhere", xdg: "xdg", source: "home/.claude/projects/sub", guarded: true},
+ {name: "in ~/.claude with both elsewhere", claude: "cfg", xdg: "xdg", source: "home/.claude/plugins/data", guarded: true},
+ {name: "in XDG_CONFIG_HOME/claude with CLAUDE_CONFIG_DIR elsewhere", claude: "cfg", xdg: "xdg", source: "xdg/claude/projects", guarded: true},
+ {name: "a link to ~/.claude/projects", claude: "cfg", xdg: "xdg", source: "scenario/_source", link: "home/.claude/projects", guarded: true},
+ {name: "a place of its own", claude: "cfg", xdg: "xdg", source: "scenario/_source"},
+ } {
+ t.Run(c.name, func(t *testing.T) {
+ if c.link != "" && runtime.GOOS == "windows" {
+ t.Skip("a symbolic link needs a privilege on Windows")
+ }
+ base := t.TempDir()
+ at := func(rel string) string {
+ if rel == "" {
+ return ""
+ }
+ return filepath.Join(base, filepath.FromSlash(rel))
+ }
+ t.Setenv("CLAUDE_CONFIG_DIR", at(c.claude))
+ t.Setenv("XDG_CONFIG_HOME", at(c.xdg))
+ t.Setenv("HOME", at("home"))
+ t.Setenv("USERPROFILE", at("home"))
+ source := at(c.source)
+ if c.link != "" {
+ if err := os.MkdirAll(at(c.link), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.MkdirAll(filepath.Dir(source), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(at(c.link), source); err != nil {
+ t.Fatal(err)
+ }
+ }
+ kept := filepath.Join(source, "-Users-dev-real", id+".jsonl")
+ if err := os.MkdirAll(filepath.Dir(kept), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.WriteFile(kept, []byte("{}\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := scenario.WriteMarker(scenario.MarkerPath(source, id), &scenario.Marker{Schema: 1, Session: id,
+ Policy: scenario.PolicyImmediately, State: scenario.MarkerRemoving, Files: []scenario.MarkerFile{}}); err != nil {
+ t.Fatal(err)
+ }
+ zone := storage.NewZone(t.TempDir())
+ changes := filepath.Join(source, "plugins", "data")
+ rm := &remove.Remover{Zone: zone, Source: source, Local: claudecode.New(source, zone, 0),
+ Changes: claudecodechanges.New(changes, zone, 0), ChangesRoot: changes}
+ res := rm.Pass(remove.Input{Endpoint: "grpc://127.0.0.1:11800", SendLogs: true, SendMetrics: true})
+ noErrors(t, res)
+ if got := keptFor(res, "", "where Claude Code keeps its own files"); got != c.guarded {
+ t.Fatalf("guarded %v, want %v: kept %v, waiting %v", got, c.guarded, res.Kept, res.Waiting)
+ }
+ if !c.guarded {
+ // The guard let it through, so the pass went on and read what
+ // was sent. Nothing was, so it waits.
+ waited := false
+ for _, w := range res.Waiting {
+ waited = waited || w.Reason == "not sent yet"
+ }
+ if !waited {
+ t.Fatalf("the pass stopped before it read what was sent: kept %v, waiting %v", res.Kept, res.Waiting)
+ }
+ }
+ if len(res.Removed) != 0 {
+ t.Fatalf("removed %v", res.Removed)
+ }
+ if !exists(kept) || !exists(scenario.MarkerPath(source, id)) {
+ t.Fatal("something was deleted under the source directory")
+ }
+ })
+ }
+}
+
+// TestALinkAtRemovedKeepsEverySession. A _removed that is a symbolic link
+// leads out of the root. The pass stops before any session reaches step 0,
+// so none is left half removed while a person looks at it, and nothing the
+// link points at is touched.
+func TestALinkAtRemovedKeepsEverySession(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("a symbolic link needs a privilege on Windows")
+ }
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ r.pipeline()
+ outside := t.TempDir()
+ precious := filepath.Join(outside, "precious.jsonl")
+ if err := os.WriteFile(precious, []byte("{}\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(outside, filepath.Join(r.dir, storage.RemovedDir)); err != nil {
+ t.Fatal(err)
+ }
+ res := r.remover().Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !keptFor(res, "", "is a symbolic link or not a directory") {
+ t.Fatalf("removed %v, kept %v", res.Removed, res.Kept)
+ }
+ m := marker(t, b)
+ if m.State != scenario.MarkerWritten {
+ t.Fatalf("the marker says %q; no session may reach step 0 while %s is a link", m.State, storage.RemovedDir)
+ }
+ for _, f := range m.Files {
+ if !exists(filepath.Join(r.source, filepath.FromSlash(f.Path))) {
+ t.Fatalf("the source file %s was deleted", f.Path)
+ }
+ }
+ if !exists(precious) || !exists(filepath.Join(r.dir, b.Session)) {
+ t.Fatal("the file behind the link, or the session directory, was deleted")
+ }
+}
+
+// TestALinkOutOfTheSourceStopsTheRemoval. A project directory can be a
+// symbolic link to somewhere else. A removal must never delete through it,
+// so it stops, and the session is left for a person.
+func TestALinkOutOfTheSourceStopsTheRemoval(t *testing.T) {
+ if runtime.GOOS == "windows" {
+ t.Skip("a symbolic link needs a privilege on Windows")
+ }
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ r.pipeline()
+ // As if a pass had decided, so the checks inside the steps are what
+ // meet the link.
+ m := marker(t, b)
+ m.State = scenario.MarkerRemoving
+ if err := scenario.WriteMarker(b.Marker, m); err != nil {
+ t.Fatal(err)
+ }
+ project := filepath.Join(r.source, "-Users-dev-scenario")
+ outside := filepath.Join(t.TempDir(), "-Users-dev-scenario")
+ if err := os.Rename(project, outside); err != nil {
+ t.Fatal(err)
+ }
+ if err := os.Symlink(outside, project); err != nil {
+ t.Fatal(err)
+ }
+ before := tree(t, outside)
+
+ rm := r.remover()
+ res := rm.Pass(r.input())
+ if len(res.Errors) != 1 || !strings.Contains(res.Errors[0].Error(), "the removal stopped") {
+ t.Fatalf("errors %v; want the removal to stop", res.Errors)
+ }
+ if after := tree(t, outside); !equal(before, after) {
+ t.Fatalf("the files the link leads to changed:\nbefore %v\nafter %v", before, after)
+ }
+ if again := marker(t, b); again.State != scenario.MarkerRemoving {
+ t.Fatalf("the marker says %s; a stopped removal leaves it removing", again.State)
+ }
+ // Said once. The session waits for a person, not for the next pass.
+ if res := rm.Pass(r.input()); len(res.Errors) != 0 || len(res.Removed) != 0 {
+ t.Fatalf("the next pass gave errors %v and removed %v", res.Errors, res.Removed)
+ }
+}
+
+// tree lists every file under dir with its content.
+func tree(t *testing.T, dir string) map[string]string {
+ t.Helper()
+ out := map[string]string{}
+ err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+ if err != nil || d.IsDir() {
+ return err
+ }
+ data, err := os.ReadFile(p)
+ out[p] = string(data)
+ return err
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return out
+}
+
+func equal(a, b map[string]string) bool {
+ if len(a) != len(b) {
+ return false
+ }
+ for k, v := range a {
+ if b[k] != v {
+ return false
+ }
+ }
+ return true
+}
+
+// TestAMarkerARemovalCannotTrustKeepsItsSession. A marker decides what is
+// deleted, so one that names a path out of the source directory, or another
+// session than its own, keeps its session. A file the build is still writing
+// is not a marker at all.
+func TestAMarkerARemovalCannotTrustKeepsItsSession(t *testing.T) {
+ r := newRoot(t)
+ const id, other = "c30736f2-ac0c-4a72-89b1-01a0844af62d", "aaaaaaaa-bbbb-4ccc-8ddd-000000000001"
+ dir := filepath.Join(r.source, scenario.MarkerDir)
+ write := func(name string, m *scenario.Marker) {
+ if err := scenario.WriteMarker(filepath.Join(dir, name), m); err != nil {
+ t.Fatal(err)
+ }
+ }
+ file := []scenario.MarkerFile{{Path: "../outside.jsonl", SHA256: strings.Repeat("0", 64)}}
+ write(id+".json", &scenario.Marker{Schema: 1, Session: id, Policy: scenario.PolicyImmediately, State: scenario.MarkerWritten, Files: file})
+ write(other+".json", &scenario.Marker{Schema: 1, Session: id, Policy: scenario.PolicyImmediately, State: scenario.MarkerWritten, Files: []scenario.MarkerFile{}})
+ if err := os.WriteFile(filepath.Join(dir, ".tmp-123456"), []byte("{"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ // A receiver is recorded, so the pass gets as far as the markers.
+ state := fmt.Sprintf("schema 1\nupdated_at 2026-01-01T00:00:00Z\nendpoint %s\n", r.endpoint)
+ if err := os.WriteFile(filepath.Join(r.dir, otlp.StateFile), []byte(state), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ res := r.remover().Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || len(res.Kept) != 2 ||
+ !keptFor(res, id, "not inside the source directory") || !keptFor(res, other, "names the session") {
+ t.Fatalf("removed %v, kept %v", res.Removed, res.Kept)
+ }
+}
+
+// TestALandedFileFromAnotherSourceKeepsTheSession. A real session whose id a
+// scenario reused could land beside the scenario's files. Every landed file
+// has to come from a file the build wrote, or none of the session goes.
+func TestALandedFileFromAnotherSourceKeepsTheSession(t *testing.T) {
+ r := newRoot(t)
+ b := r.build(load(t, everyKind), "", scenario.Options{})
+ m := marker(t, b)
+ var child string
+ for _, f := range m.Files {
+ if strings.Contains(f.Path, "/subagents/agent-") && strings.HasSuffix(f.Path, ".jsonl") && !strings.Contains(f.Path, "/workflows/") {
+ child = filepath.Join(r.source, filepath.FromSlash(f.Path))
+ }
+ }
+ if child == "" {
+ t.Fatal("the scenario wrote no child transcript")
+ }
+ data, err := os.ReadFile(child)
+ if err != nil {
+ t.Fatal(err)
+ }
+ foreign := filepath.Join(filepath.Dir(child), "agent-a0123456789abcdef.jsonl")
+ if err := os.WriteFile(foreign, data, 0o600); err != nil {
+ t.Fatal(err)
+ }
+ r.collect()
+ if err := os.Remove(foreign); err != nil {
+ t.Fatal(err)
+ }
+ res := r.remover().Pass(r.input())
+ noErrors(t, res)
+ // Nothing is sent yet, so the pass stops at the first condition every
+ // session shares. Record a receiver, as a push would, and look again.
+ if !waitingFor(res, "", "not sent yet") {
+ t.Fatalf("waiting %v", res.Waiting)
+ }
+ state := fmt.Sprintf("schema 1\nupdated_at 2026-01-01T00:00:00Z\nendpoint %s\n", r.endpoint)
+ if err := os.WriteFile(filepath.Join(r.dir, otlp.StateFile), []byte(state), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ res = r.remover().Pass(r.input())
+ noErrors(t, res)
+ if len(res.Removed) != 0 || !keptFor(res, b.Session, "names a source the build did not write") {
+ t.Fatalf("removed %v, kept %v, waiting %v", res.Removed, res.Kept, res.Waiting)
+ }
+}
+
+// TestTheCapLeavesTheRestForTheNextPass. A removal holds the export lock, so
+// one pass takes at most MaxPerPass sessions, and the next takes the rest.
+func TestTheCapLeavesTheRestForTheNextPass(t *testing.T) {
+ r := newRoot(t)
+ sc, err := scenario.Load(filepath.Join("..", "..", "..", "tests", "scenarios", "assembly.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ const n = 20
+ for i := range n {
+ r.build(sc, fmt.Sprintf("aaaaaaaa-bbbb-4ccc-8ddd-%012d", i+1), scenario.Options{At: at.Add(time.Duration(i) * time.Hour)})
+ }
+ r.pipeline()
+ rm := r.remover()
+ first := rm.Pass(r.input())
+ noErrors(t, first)
+ second := rm.Pass(r.input())
+ noErrors(t, second)
+ if len(first.Removed) != remove.MaxPerPass || len(second.Removed) != n-remove.MaxPerPass {
+ t.Fatalf("the passes removed %d and %d, want %d and %d", len(first.Removed), len(second.Removed), remove.MaxPerPass, n-remove.MaxPerPass)
+ }
+}
diff --git a/internal/scenario/run/remove.go b/internal/scenario/run/remove.go
new file mode 100644
index 0000000..f08accc
--- /dev/null
+++ b/internal/scenario/run/remove.go
@@ -0,0 +1,696 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package run
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path"
+ "path/filepath"
+ "strings"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecodechanges"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp/otlptest"
+ "github.com/apache/skywalking-ai-sessionizer/internal/metrics"
+ "github.com/apache/skywalking-ai-sessionizer/internal/parse"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario/remove"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+ "github.com/apache/skywalking-ai-sessionizer/internal/view"
+)
+
+// elsewhere names a receiver no check sends to. Nothing listens there, and
+// nothing tries to reach it. Only the removal compares it with push.state.
+const elsewhere = "grpc://another-receiver.invalid:4317"
+
+// retention is the policy the retention case writes into its copy's marker.
+const retention = 24 * time.Hour
+
+// removal is one run of the removed_after_sent property over one format. It
+// only reads the finished root. Every case works on a full copy of it, and
+// every copy sends to the one receiver the run starts.
+type removal struct {
+ sc *scenario.Scenario
+ format scenario.Format
+ out string
+ session string
+ project string
+ maxRound int64
+ opts Options
+ rcv *otlptest.Receiver
+ client otlp.Client
+ endpoint string
+ copies int
+ problems []string
+}
+
+// removedAfterSent checks what a pipeline's removal does with a scenario
+// session. Nothing goes before all of it is recorded as sent to the receiver
+// the pipeline sends to. All of it goes once it is, by the policy in its
+// marker, and nothing is landed, derived, parsed or sent again afterwards. A
+// session with no marker, which every real Claude Code session is, stays.
+func removedAfterSent(sc *scenario.Scenario, f scenario.Format, out, session, project string, maxRound int64, opts Options) ([]string, error) {
+ rcv, err := otlptest.Start()
+ if err != nil {
+ return nil, err
+ }
+ defer rcv.Close()
+ o := rcv.Options(otlp.ProtocolGRPC)
+ client, err := otlp.NewClient(o)
+ if err != nil {
+ return nil, err
+ }
+ defer client.Close()
+ x := &removal{sc: sc, format: f, out: out, session: session, project: project, maxRound: maxRound, opts: opts,
+ rcv: rcv, client: client, endpoint: otlp.EndpointOf(o.Protocol, o.Endpoint, o.TLS)}
+ type step struct {
+ name string
+ fn func(*removalCase) error
+ }
+ cases := []step{
+ {"before any push", (*removalCase).beforeAnyPush},
+ {"a refused push", (*removalCase).refused},
+ {"a partial success", (*removalCase).rejected},
+ {"another endpoint", (*removalCase).anotherEndpoint},
+ {"no marker", (*removalCase).noMarker},
+ {"immediately", (*removalCase).immediately},
+ {"a retention of " + retention.String(), (*removalCase).retained},
+ }
+ if f != scenario.FormatClaudeCode {
+ cases = []step{{"an sd build", (*removalCase).neverMarked}}
+ }
+ for _, c := range cases {
+ if err := x.run(c.name, c.fn); err != nil {
+ return nil, fmt.Errorf("removed_after_sent: %s: %w", c.name, err)
+ }
+ }
+ return x.problems, nil
+}
+
+// run makes a fresh copy of the finished root and runs one case on it. The
+// copy's push.state is deleted first: push_follows_the_wire leaves one
+// behind, and a case must start with nothing sent. The receiver starts each
+// case empty and taking everything.
+func (x *removal) run(name string, fn func(*removalCase) error) error {
+ x.copies++
+ dir := fmt.Sprintf("%s-removal-%d", x.out, x.copies)
+ defer os.RemoveAll(dir)
+ if err := copyRoot(x.out, dir); err != nil {
+ return err
+ }
+ if err := os.Remove(filepath.Join(dir, otlp.StateFile)); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ x.rcv.Fail(false)
+ x.rcv.Reject(0)
+ x.rcv.Reset()
+ c := &removalCase{removal: x, name: name, dir: dir, source: filepath.Join(dir, "_source"), zone: storage.NewZone(dir)}
+ c.rm = c.remover()
+ return fn(c)
+}
+
+// removalCase is one case, on its own copy of the root.
+type removalCase struct {
+ *removal
+ name string
+ dir string
+ source string
+ zone *storage.Zone
+ rm *remove.Remover
+}
+
+func (c *removalCase) bad(format string, a ...any) {
+ c.problems = append(c.problems, "removed_after_sent: "+c.name+": "+fmt.Sprintf(format, a...))
+}
+
+func (c *removalCase) changesRoot() string { return filepath.Join(c.source, "plugins", "data") }
+func (c *removalCase) markerPath() string { return scenario.MarkerPath(c.source, c.session) }
+func (c *removalCase) chainDir() string { return filepath.Join(c.dir, "_conversations", c.session) }
+
+func (c *removalCase) sourcePath(rel string) string {
+ return filepath.Join(c.source, filepath.FromSlash(rel))
+}
+
+// rel is a path under the copy as push.state writes it.
+func (c *removalCase) rel(p string) string {
+ rel, err := filepath.Rel(c.dir, p)
+ if err != nil {
+ return p
+ }
+ return filepath.ToSlash(rel)
+}
+
+// remover is the removal a pipeline over the copy runs, as the build's
+// configuration sets the pipeline up: both adapters, the changes adapter
+// reading where the build wrote the plugin's output, and metrics derived.
+func (c *removalCase) remover() *remove.Remover {
+ return &remove.Remover{Zone: c.zone, Source: c.source,
+ Local: claudecode.New(c.source, c.zone, 0),
+ Changes: claudecodechanges.New(c.changesRoot(), c.zone, 0), ChangesRoot: c.changesRoot(),
+ Derives: true, Now: checkNow}
+}
+
+// push sends what the copy has not sent yet, and records the check's
+// receiver in push.state, as the pipeline's pusher does.
+func (c *removalCase) push() (*otlp.Stats, error) {
+ p := &otlp.Pusher{Zone: c.zone, Client: c.client, Endpoint: c.endpoint, Version: "check",
+ ServiceName: "Scenario Check", InstanceID: "scenario-check", Layer: "AI_AGENT", Now: checkNow}
+ return p.Pass()
+}
+
+// prepare is a pass of the pipeline up to the push: it lands with both
+// adapters, derives, and parses every session directory.
+func (c *removalCase) prepare() error {
+ if _, err := collect(c.format, c.dir); err != nil {
+ return err
+ }
+ if _, err := derive(c.dir); err != nil {
+ return err
+ }
+ _, err := parseEvery(c.dir, c.maxRound)
+ return err
+}
+
+// send is a pass of the pipeline up to the removal, to a receiver that
+// takes everything.
+func (c *removalCase) send() (*otlp.Stats, error) {
+ if err := c.prepare(); err != nil {
+ return nil, err
+ }
+ st, err := c.push()
+ if err != nil {
+ return nil, err
+ }
+ if len(st.Errors) != 0 {
+ return nil, fmt.Errorf("push: %v", st.Errors)
+ }
+ return st, nil
+}
+
+// removeNow is the last act of a pass. An error the removal reports is a
+// problem of the property, so the other cases still run.
+func (c *removalCase) removeNow(endpoint string) *remove.Result {
+ res := c.rm.Pass(remove.Input{Endpoint: endpoint, SendLogs: true, SendMetrics: true})
+ for _, err := range res.Errors {
+ c.bad("the removal reported %v", err)
+ }
+ return res
+}
+
+// beforeAnyPush. Nothing is sent, so nothing is proved sent. The marker
+// stays as the build wrote it, since a removal begins by rewriting it.
+func (c *removalCase) beforeAnyPush() error {
+ if err := c.prepare(); err != nil {
+ return err
+ }
+ res := c.removeNow(c.endpoint)
+ if len(res.Removed) != 0 {
+ c.bad("removed %v before anything was sent", res.Removed)
+ }
+ m, err := scenario.ReadMarker(c.markerPath())
+ if err != nil {
+ return err
+ }
+ if m.State != scenario.MarkerWritten {
+ c.bad("the marker says %s after a pass that removed nothing, not %s", m.State, scenario.MarkerWritten)
+ }
+ return nil
+}
+
+// refused. A receiver that refuses a request holds none of it. Nothing is
+// recorded as sent, so nothing is removed.
+func (c *removalCase) refused() error {
+ if err := c.prepare(); err != nil {
+ return err
+ }
+ c.rcv.Fail(true)
+ st, err := c.push()
+ c.rcv.Fail(false)
+ if err != nil {
+ return err
+ }
+ if st.Files != 0 || len(st.Errors) == 0 {
+ c.bad("a push the receiver refused marked %d file(s) sent and reported %d error(s)", st.Files, len(st.Errors))
+ }
+ if res := c.removeNow(c.endpoint); len(res.Removed) != 0 {
+ c.bad("removed %v after the receiver refused every request", res.Removed)
+ }
+ return nil
+}
+
+// rejected. A receiver that took a request and rejected records of it holds
+// less than was sent. The session stays, and the pass says why. That record
+// has to survive every later save of push.state, or the session would go at
+// the next pass: a second session, sent whole, makes one.
+func (c *removalCase) rejected() error {
+ c.rcv.Reject(1)
+ st, err := c.send()
+ c.rcv.Reject(0)
+ if err != nil {
+ return err
+ }
+ if st.Files == 0 {
+ c.bad("a partial success marked no file sent; OTLP says not to send such a request again")
+ }
+ res := c.removeNow(c.endpoint)
+ if len(res.Removed) != 0 || !says(res.Kept, c.session, "rejected records") {
+ c.bad("after a partial success: removed %v, kept %v; the session must stay, and the pass must say the receiver rejected records", res.Removed, res.Kept)
+ }
+
+ other := *c.sc
+ other.Session = anotherSession(c.session)
+ b, err := scenario.Build(&other, scenario.FormatClaudeCode, c.dir, scenario.Options{At: c.opts.At, Scale: c.opts.Scale, Interval: c.opts.Interval})
+ if err != nil {
+ return err
+ }
+ if st, err = c.send(); err != nil {
+ return err
+ }
+ if st.Files == 0 {
+ c.bad("the second session sent nothing, so push.state was not saved again")
+ }
+ res = c.removeNow(c.endpoint)
+ if len(res.Removed) != 1 || res.Removed[0] != b.Session {
+ c.bad("removed %v; want only %s, the second session, which was sent whole", res.Removed, b.Session)
+ }
+ if !present(c.markerPath()) || !present(c.zone.SessionDir(c.session)) {
+ c.bad("the session whose records were rejected was removed after a later save of push.state")
+ }
+ sent, err := otlp.LoadSent(c.dir)
+ if err != nil {
+ return err
+ }
+ landed, err := storage.LandedFiles(c.zone, c.session)
+ if err != nil {
+ return err
+ }
+ for _, lf := range landed {
+ if rel := c.rel(lf.Path); !sent.Rejected(rel) {
+ c.bad("push.state no longer records that the receiver rejected records of %s", rel)
+ }
+ }
+ return nil
+}
+
+// anotherEndpoint. push.state records which receivers files went to, not
+// which file went to which. Sent somewhere other than where this pipeline
+// sends, nothing is proved sent to it, and every session stays.
+func (c *removalCase) anotherEndpoint() error {
+ if _, err := c.send(); err != nil {
+ return err
+ }
+ res := c.removeNow(elsewhere)
+ if len(res.Removed) != 0 || !says(res.Kept, "", "push.state records sending to "+c.endpoint) {
+ c.bad("sent to %s and removing for %s: removed %v, kept %v", c.endpoint, elsewhere, res.Removed, res.Kept)
+ }
+ if !present(c.markerPath()) {
+ c.bad("the marker was deleted")
+ }
+ return nil
+}
+
+// noMarker. A session no build marked, as every real Claude Code session
+// is, stays whatever was sent. Nothing of it is touched, and nothing is said
+// about it.
+func (c *removalCase) noMarker() error {
+ m, err := scenario.ReadMarker(c.markerPath())
+ if err != nil {
+ return err
+ }
+ if err := os.Remove(c.markerPath()); err != nil {
+ return err
+ }
+ if _, err := c.send(); err != nil {
+ return err
+ }
+ res := c.removeNow(c.endpoint)
+ if len(res.Removed) != 0 || len(res.Kept) != 0 {
+ c.bad("with no marker: removed %v, kept %v", res.Removed, res.Kept)
+ }
+ for _, f := range m.Files {
+ if !present(c.sourcePath(f.Path)) {
+ c.bad("the source file %s was deleted", f.Path)
+ }
+ }
+ for _, p := range []string{c.zone.SessionDir(c.session), c.chainDir()} {
+ if !present(p) {
+ c.bad("%s was deleted", c.rel(p))
+ }
+ }
+ return nil
+}
+
+// immediately. Once all of the session is sent, one pass removes all of it,
+// and nothing that other sessions share. Afterwards nothing lands, derives,
+// parses or sends it again, and a second removal finds nothing to do.
+func (c *removalCase) immediately() error {
+ m, err := scenario.ReadMarker(c.markerPath())
+ if err != nil {
+ return err
+ }
+ if _, err := c.send(); err != nil {
+ return err
+ }
+ res := c.removeNow(c.endpoint)
+ if len(res.Removed) != 1 || res.Removed[0] != c.session {
+ c.bad("removed %v, kept %v, waiting %v; the session was sent whole and must go", res.Removed, res.Kept, res.Waiting)
+ return nil
+ }
+ if err := c.goneWhole(m, "after the removal"); err != nil {
+ return err
+ }
+
+ c.rcv.Reset()
+ landed, err := collect(c.format, c.dir)
+ if err != nil {
+ return err
+ }
+ if landed != 0 {
+ c.bad("afterwards the adapters landed %d source(s)", landed)
+ }
+ derived, err := derive(c.dir)
+ if err != nil {
+ return err
+ }
+ if derived != 0 {
+ c.bad("afterwards the deriver wrote %d request(s)", derived)
+ }
+ rounds, err := parseEvery(c.dir, c.maxRound)
+ if err != nil {
+ return err
+ }
+ if rounds != 0 {
+ c.bad("afterwards the parse wrote %d round(s)", rounds)
+ }
+ st, err := c.push()
+ if err != nil {
+ return err
+ }
+ if len(st.Errors) != 0 {
+ return fmt.Errorf("push: %v", st.Errors)
+ }
+ if n := len(c.rcv.Requests()) + len(c.rcv.MetricsRequests()); n != 0 {
+ c.bad("afterwards the push sent %d request(s)", n)
+ }
+ if again := c.removeNow(c.endpoint); len(again.Removed) != 0 {
+ c.bad("a second removal removed %v", again.Removed)
+ }
+ return c.goneWhole(m, "afterwards")
+}
+
+// retained. A retained session stays until its last record is as old as
+// the policy says. The age counts from the records, never from when anything
+// ran, so the same data decides the same way.
+func (c *removalCase) retained() error {
+ m, err := scenario.ReadMarker(c.markerPath())
+ if err != nil {
+ return err
+ }
+ m.Policy, m.Retain = scenario.PolicyRetain, retention.String()
+ if err := scenario.WriteMarker(c.markerPath(), m); err != nil {
+ return err
+ }
+ last, err := lastSourceTime(c.source, m)
+ if err != nil {
+ c.bad("%v", err)
+ return nil
+ }
+ if _, err := c.send(); err != nil {
+ return err
+ }
+ now := last.Add(retention - time.Second)
+ c.rm.Now = func() time.Time { return now }
+ res := c.removeNow(c.endpoint)
+ if len(res.Removed) != 0 || !says(res.Waiting, c.session, "kept until") {
+ c.bad("a second before its last record, at %s, is %s old: removed %v, waiting %v; it must wait", last.Format(time.RFC3339Nano), retention, res.Removed, res.Waiting)
+ }
+ // The same remover, so what it remembers of the last record is used too.
+ now = last.Add(retention)
+ res = c.removeNow(c.endpoint)
+ if len(res.Removed) != 1 || res.Removed[0] != c.session {
+ c.bad("once its last record, at %s, is %s old: removed %v, waiting %v, kept %v; it must go", last.Format(time.RFC3339Nano), retention, res.Removed, res.Waiting, res.Kept)
+ return nil
+ }
+ return c.goneWhole(m, "after the retention")
+}
+
+// neverMarked. An sd build writes no source, so it writes no marker and no
+// sign of a scenario root, and nothing of it is removed.
+func (c *removalCase) neverMarked() error {
+ for _, p := range []string{filepath.Join(c.source, scenario.MarkerDir), filepath.Join(c.dir, storage.ScenarioDir)} {
+ if present(p) {
+ c.bad("an sd build left %s", c.rel(p))
+ }
+ }
+ if _, err := c.send(); err != nil {
+ return err
+ }
+ res := c.removeNow(c.endpoint)
+ if len(res.Removed) != 0 || len(res.Kept) != 0 {
+ c.bad("removed %v, kept %v", res.Removed, res.Kept)
+ }
+ for _, p := range []string{c.zone.SessionDir(c.session), c.chainDir()} {
+ if !present(p) {
+ c.bad("%s was deleted", c.rel(p))
+ }
+ }
+ return nil
+}
+
+// goneWhole checks that nothing of the session is left, and that nothing
+// other sessions share went with it.
+func (c *removalCase) goneWhole(m *scenario.Marker, when string) error {
+ bad := func(format string, a ...any) { c.bad(when+": "+format, a...) }
+ s := c.session
+ if present(c.markerPath()) {
+ bad("the marker is still there")
+ }
+ shared := map[string]bool{}
+ for _, f := range m.Files {
+ if present(c.sourcePath(f.Path)) {
+ bad("the source file %s is still there", f.Path)
+ }
+ above, own := sessionDirOf(f.Path, s)
+ if own != "" && present(c.sourcePath(own)) {
+ bad("the directory %s, named for the session, is still there", own)
+ }
+ shared[above] = true
+ }
+ for _, dir := range keys(shared) {
+ if !present(c.sourcePath(dir)) {
+ bad("%s, which other sessions share, was removed", dir)
+ }
+ }
+ project := filepath.Join(c.source, filepath.FromSlash(c.project))
+ for _, p := range []string{
+ filepath.Join(project, "not-a-uuid.jsonl"),
+ filepath.Join(project, "memory", "notes.md"),
+ filepath.Join(c.source, scenario.MarkerDir),
+ filepath.Join(c.dir, storage.ScenarioDir),
+ } {
+ if !present(p) {
+ bad("%s, which every session shares, was removed", c.rel(p))
+ }
+ }
+ for _, p := range []string{c.zone.SessionDir(s), c.chainDir()} {
+ if present(p) {
+ bad("%s is still there", c.rel(p))
+ }
+ }
+ spool, err := os.ReadDir(filepath.Join(c.dir, storage.SpoolDir))
+ if err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ for _, it := range spool {
+ if owner, ok := metrics.SpoolOwner(it.Name()); ok && owner == s {
+ bad("the spool file %s is still there", it.Name())
+ }
+ }
+ for _, state := range []string{filepath.Join(c.dir, otlp.StateFile), filepath.Join(c.dir, storage.SpoolDir, metrics.StateFile)} {
+ data, err := os.ReadFile(state)
+ if err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ if bytes.Contains(data, []byte(s)) {
+ bad("%s still names the session", c.rel(state))
+ }
+ }
+ left, err := os.ReadDir(filepath.Join(c.dir, storage.RemovedDir))
+ if err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ if len(left) != 0 {
+ bad("%s still holds %d entries", storage.RemovedDir, len(left))
+ }
+ ids, err := view.New(c.zone, nil).List()
+ if err != nil {
+ return err
+ }
+ for _, id := range ids {
+ if id == s {
+ bad("the page still lists the conversation")
+ }
+ }
+ return nil
+}
+
+// sessionDirOf splits a marker path at its first segment equal to the
+// session. It returns the directory above that segment, which other sessions
+// share, and the directory the segment names, which is the session's own. A
+// path with no such segment, such as the main transcript, has no directory
+// of its own, and its parent is shared.
+func sessionDirOf(p, session string) (shared, own string) {
+ parts := strings.Split(p, "/")
+ for i := 0; i < len(parts)-1; i++ {
+ if parts[i] == session {
+ return strings.Join(parts[:i], "/"), strings.Join(parts[:i+1], "/")
+ }
+ }
+ return path.Dir(p), ""
+}
+
+// lastSourceTime is the latest time the session's source lines carry, read
+// as each producer writes it: timestamp in Claude Code's own files, and time
+// in the plugin's. It reads no landed file, so it checks the removal's
+// reading of the landed records rather than repeat it.
+func lastSourceTime(source string, m *scenario.Marker) (time.Time, error) {
+ var last time.Time
+ for _, f := range m.Files {
+ key := "timestamp"
+ if strings.HasPrefix(f.Path, "plugins/data/") {
+ key = "time"
+ }
+ data, err := os.ReadFile(filepath.Join(source, filepath.FromSlash(f.Path)))
+ if err != nil {
+ return last, err
+ }
+ for _, line := range bytes.Split(data, []byte("\n")) {
+ var fields map[string]json.RawMessage
+ if json.Unmarshal(line, &fields) != nil {
+ continue
+ }
+ var s string
+ if json.Unmarshal(fields[key], &s) != nil {
+ continue
+ }
+ if t, err := time.Parse(time.RFC3339Nano, s); err == nil && t.After(last) {
+ last = t
+ }
+ }
+ }
+ if last.IsZero() {
+ return last, errors.New("no line of the session's source files carries a time, so no retention can be counted")
+ }
+ return last, nil
+}
+
+// parseEvery parses every session directory of a root, as a pass of the
+// pipeline does, and returns how many rounds it wrote.
+func parseEvery(root string, maxRound int64) (int, error) {
+ items, err := os.ReadDir(root)
+ if err != nil {
+ return 0, err
+ }
+ z := storage.NewZone(root)
+ written := 0
+ for _, it := range items {
+ if !it.IsDir() || strings.HasPrefix(it.Name(), "_") {
+ continue
+ }
+ for {
+ r, err := parse.Session(z, parse.Options{Conversation: it.Name(), Session: it.Name(), MaxRoundBytes: maxRound})
+ if err != nil {
+ return written, fmt.Errorf("parse %s: %w", it.Name(), err)
+ }
+ if r.Changed() {
+ written++
+ }
+ if !r.Changed() || !r.More {
+ break
+ }
+ }
+ }
+ return written, nil
+}
+
+// copyRoot copies a root, modes kept. Nothing in a root records where the
+// root is, so the copy is a root in its own right. The configuration is left
+// out, because a build into the copy refuses one that names another
+// directory.
+func copyRoot(from, to string) error {
+ return filepath.WalkDir(from, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ rel, err := filepath.Rel(from, p)
+ if err != nil {
+ return err
+ }
+ if rel == "asz.yaml" {
+ return nil
+ }
+ info, err := d.Info()
+ if err != nil {
+ return err
+ }
+ target := filepath.Join(to, rel)
+ if d.IsDir() {
+ return os.MkdirAll(target, info.Mode().Perm())
+ }
+ if !info.Mode().IsRegular() {
+ return fmt.Errorf("%s is not a regular file, and a root holds nothing else", p)
+ }
+ data, err := os.ReadFile(p)
+ if err != nil {
+ return err
+ }
+ return os.WriteFile(target, data, info.Mode().Perm())
+ })
+}
+
+// anotherSession is a session id that is not the scenario's own.
+func anotherSession(session string) string {
+ id := "aaaaaaaa-bbbb-4ccc-8ddd-000000000001"
+ if strings.EqualFold(id, session) {
+ id = "aaaaaaaa-bbbb-4ccc-8ddd-000000000002"
+ }
+ return id
+}
+
+// says reports whether a list of kept or waiting sessions holds one with a
+// reason that contains the words given. An empty session is every session.
+func says(list []remove.Kept, session, reason string) bool {
+ for _, k := range list {
+ if k.Session == session && strings.Contains(k.Reason, reason) {
+ return true
+ }
+ }
+ return false
+}
+
+func present(p string) bool {
+ _, err := os.Lstat(p)
+ return err == nil
+}
diff --git a/internal/scenario/run/run.go b/internal/scenario/run/run.go
index 5a036ca..e1776bc 100644
--- a/internal/scenario/run/run.go
+++ b/internal/scenario/run/run.go
@@ -160,12 +160,12 @@ func checkFormat(sc *scenario.Scenario, ex *expect.File, f scenario.Format, out
if err != nil {
return nil, "", err
}
- if err := collect(f, out); err != nil {
+ if _, err := collect(f, out); err != nil {
return nil, "", err
}
// The runtime's metric family, derived as the collector derives it,
// after each landing and before anything later happens to the root.
- if err := derive(out); err != nil {
+ if _, err := derive(out); err != nil {
return nil, "", err
}
round, err = parseAll(out, built.Session, ex.Parse.MaxRoundBytes)
@@ -245,6 +245,9 @@ func checkFormat(sc *scenario.Scenario, ex *expect.File, f scenario.Format, out
{"push_follows_the_wire", ex.Properties.PushFollowsTheWire, func() ([]string, error) {
return pushFollowsTheWire(out, session, f, ex.Push, expectedTokens(built.Plan), expect.On(ex.Properties.MetricsMatchThePlan))
}},
+ {"removed_after_sent", ex.Properties.RemovedAfterSent, func() ([]string, error) {
+ return removedAfterSent(sc, f, out, session, built.Plan.Project, ex.Parse.MaxRoundBytes, opts)
+ }},
{"view_covers_the_session", ex.Properties.ViewCoversTheSession, func() ([]string, error) { return expect.ViewCoversTheSession(out, session) }},
{"changes_leave_the_fold", ex.Properties.ChangesLeaveTheFold, func() ([]string, error) {
if !scenario.HasChanges(sc.Steps) {
@@ -352,41 +355,47 @@ func lose(out, session string, l expect.Lose, ctx *expect.Context) (string, erro
}
// derive writes the runtime's metric family for what is landed and not yet
-// derived, with no look-back: a scenario is history by construction.
-func derive(out string) error {
- d := &metrics.Deriver{Zone: storage.NewZone(out), Options: metrics.Options{Grace: -1, Version: "check", Now: func() time.Time { return time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC) }}}
+// derived, with no look-back: a scenario is history by construction. It
+// returns how many requests it put in the spool.
+func derive(out string) (int, error) {
+ d := &metrics.Deriver{Zone: storage.NewZone(out), Options: metrics.Options{Grace: -1, Version: "check", Now: checkNow}}
st, err := d.Pass(nil)
if err != nil {
- return err
+ return 0, err
}
if len(st.Errors) != 0 {
- return fmt.Errorf("metrics: %v", st.Errors)
+ return 0, fmt.Errorf("metrics: %v", st.Errors)
}
- return nil
+ return st.Requests, nil
}
-// collect lands a runtime format's source through its adapter. An sd build
-// is landed already.
-func collect(f scenario.Format, out string) error {
+// checkNow is the clock of everything a check runs after the build: the
+// deriver, the pusher and the removal. It is fixed, so a check writes the
+// same bytes every time.
+func checkNow() time.Time { return time.Date(2026, 1, 3, 0, 0, 0, 0, time.UTC) }
+
+// collect lands a runtime format's source through its adapter, and returns
+// how many sources it landed from. An sd build is landed already.
+func collect(f scenario.Format, out string) (int, error) {
if f != scenario.FormatClaudeCode {
- return nil
+ return 0, nil
}
st, err := claudecode.New(filepath.Join(out, "_source"), storage.NewZone(out), 0).CollectAll(nil)
if err != nil {
- return err
+ return 0, err
}
if len(st.Errors) != 0 {
- return fmt.Errorf("collect: %v", st.Errors)
+ return 0, fmt.Errorf("collect: %v", st.Errors)
}
// The plugin's lines, through their own adapter, as asz view runs both.
cs, err := claudecodechanges.New(filepath.Join(out, "_source", "plugins", "data"), storage.NewZone(out), 0).CollectAll(nil)
if err != nil {
- return err
+ return 0, err
}
if len(cs.Errors) != 0 {
- return fmt.Errorf("collect changes: %v", cs.Errors)
+ return 0, fmt.Errorf("collect changes: %v", cs.Errors)
}
- return nil
+ return st.SourcesLanded + cs.SourcesLanded, nil
}
// changesLeaveTheFold builds the scenario again without its changes and
@@ -400,7 +409,7 @@ func changesLeaveTheFold(sc *scenario.Scenario, f scenario.Format, out, session
if err != nil {
return nil, err
}
- if err := collect(f, plain); err != nil {
+ if _, err := collect(f, plain); err != nil {
return nil, err
}
if _, err := parseAll(plain, built.Session, maxRound); err != nil {
diff --git a/internal/scenario/write_claudecode.go b/internal/scenario/write_claudecode.go
index e650350..a21c182 100644
--- a/internal/scenario/write_claudecode.go
+++ b/internal/scenario/write_claudecode.go
@@ -18,6 +18,8 @@
package scenario
import (
+ "crypto/sha256"
+ "encoding/hex"
"encoding/json"
"fmt"
"os"
@@ -33,7 +35,10 @@ import (
// file is written whole, so a build through a later checkpoint into the
// same directory is an append to what the earlier build wrote, which is
// what the collector's cursor expects of a live session.
-func writeClaudeCode(p *Plan, root string) ([]string, error) {
+//
+// It also returns every file of the session, relative to root, with the size
+// and digest of the bytes written, for the session's marker.
+func writeClaudeCode(p *Plan, root string) ([]string, []MarkerFile, error) {
proj := filepath.Join(root, p.Project)
w := &ccWriter{p: p, files: map[string][]string{}, plugin: map[string][]string{}}
lost := p.lostStreams()
@@ -45,14 +50,15 @@ func writeClaudeCode(p *Plan, root string) ([]string, error) {
continue
}
if err := w.event(e); err != nil {
- return nil, err
+ return nil, nil, err
}
}
var written []string
- putUnder := func(dir, rel string, lines []string) error {
+ marked := map[string]MarkerFile{}
+ putFile := func(dir, rel string, lines []string) (string, []byte, error) {
path := filepath.Join(dir, filepath.FromSlash(rel))
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
- return err
+ return "", nil, err
}
var buf []byte
for _, l := range lines {
@@ -60,23 +66,46 @@ func writeClaudeCode(p *Plan, root string) ([]string, error) {
buf = append(buf, '\n')
}
if err := os.WriteFile(path, buf, 0o644); err != nil {
- return err
+ return "", nil, err
}
written = append(written, path)
+ return path, buf, nil
+ }
+ // putUnder writes a file of the session and records it for the marker.
+ // The size and digest are of the bytes written, so a removal deletes a
+ // file only while it still holds exactly them.
+ putUnder := func(dir, rel string, lines []string) error {
+ path, buf, err := putFile(dir, rel, lines)
+ if err != nil {
+ return err
+ }
+ under, err := filepath.Rel(root, path)
+ if err != nil {
+ return err
+ }
+ sum := sha256.Sum256(buf)
+ under = filepath.ToSlash(under)
+ marked[under] = MarkerFile{Path: under, Size: int64(len(buf)), SHA256: hex.EncodeToString(sum[:])}
return nil
}
put := func(rel string, lines []string) error { return putUnder(proj, rel, lines) }
+ // putShared writes a file every session of the project shares. It belongs
+ // to no session, so no marker lists it and no removal deletes it.
+ putShared := func(rel string, lines []string) error {
+ _, _, err := putFile(proj, rel, lines)
+ return err
+ }
if err := put(p.Session+".jsonl", w.files["main"]); err != nil {
- return nil, err
+ return nil, nil, err
}
// Noise a real project directory holds: a file whose name is not a
// session id, and a directory that is not a session. Discovery must pass
// over both, and every scenario checks that it does.
- if err := put("not-a-uuid.jsonl", []string{`{"type":"summary","summary":"not a session"}`}); err != nil {
- return nil, err
+ if err := putShared("not-a-uuid.jsonl", []string{`{"type":"summary","summary":"not a session"}`}); err != nil {
+ return nil, nil, err
}
- if err := put("memory/notes.md", []string{"# notes", "", "not a session either"}); err != nil {
- return nil, err
+ if err := putShared("memory/notes.md", []string{"# notes", "", "not a session either"}); err != nil {
+ return nil, nil, err
}
for _, s := range p.Streams {
if s.Lost {
@@ -85,18 +114,18 @@ func writeClaudeCode(p *Plan, root string) ([]string, error) {
lines := w.files[s.ID]
if s.Batch != "" {
if err := put(fmt.Sprintf("%s/subagents/workflows/%s/agent-%s.jsonl", p.Session, s.Batch, s.ID), lines); err != nil {
- return nil, err
+ return nil, nil, err
}
continue
}
if err := put(fmt.Sprintf("%s/subagents/agent-%s.jsonl", p.Session, s.ID), lines); err != nil {
- return nil, err
+ return nil, nil, err
}
meta, _ := json.Marshal(map[string]any{
"agentType": "general-purpose", "description": s.Label, "toolUseId": s.Tool, "spawnDepth": 1,
})
if err := put(fmt.Sprintf("%s/subagents/agent-%s.meta.json", p.Session, s.ID), []string{string(meta)}); err != nil {
- return nil, err
+ return nil, nil, err
}
}
for _, r := range p.Runs {
@@ -109,31 +138,36 @@ func writeClaudeCode(p *Plan, root string) ([]string, error) {
journal = append(journal, w.rec(m, ""))
}
if err := put(fmt.Sprintf("%s/subagents/workflows/%s/journal.jsonl", p.Session, r.ID), journal); err != nil {
- return nil, err
+ return nil, nil, err
}
manifest, _ := json.Marshal(map[string]any{
"runId": r.ID, "taskId": "task-" + r.ID, "workflowName": r.Name, "status": "completed", "agentCount": len(r.Children),
})
if err := put(fmt.Sprintf("%s/workflows/%s.json", p.Session, r.ID), []string{string(manifest)}); err != nil {
- return nil, err
+ return nil, nil, err
}
scriptDir := proj
if r.ScriptProject != "" {
scriptDir = filepath.Join(root, r.ScriptProject)
}
if err := putUnder(scriptDir, fmt.Sprintf("%s/workflows/scripts/%s-%s.js", p.Session, strings.ReplaceAll(r.Name, " ", "-"), r.ID), strings.Split(r.Script, "\n")); err != nil {
- return nil, err
+ return nil, nil, err
}
}
// The plugin's output, where it keeps it: its own data directory beside
// projects, one file per stream of the session.
for stream, lines := range w.plugin {
if err := putUnder(root, PluginOutputDir+"/"+p.Session+"/"+stream+".jsonl", lines); err != nil {
- return nil, err
+ return nil, nil, err
}
}
sort.Strings(written)
- return written, nil
+ files := make([]MarkerFile, 0, len(marked))
+ for _, f := range marked {
+ files = append(files, f)
+ }
+ sort.Slice(files, func(i, j int) bool { return files[i].Path < files[j].Path })
+ return written, files, nil
}
// PluginOutputDir is where, under the source root, a Claude Code build
diff --git a/internal/storage/cursor.go b/internal/storage/cursor.go
index 0d46b68..aacc4e6 100644
--- a/internal/storage/cursor.go
+++ b/internal/storage/cursor.go
@@ -46,6 +46,9 @@ const (
CursorActive = "active"
// CursorSourceGone means the source file was deleted. This is expected, not
// an error: Claude Code prunes transcripts, and our landed deltas outlive them.
+ // That holds for every real session. The one exception is a session a
+ // scenario build marked: once all of it is sent, a pipeline removes its
+ // source and then its whole session directory, this cursor included.
CursorSourceGone = "source_gone"
// CursorConflict means the source changed underneath us - rotated, truncated,
// or rewritten behind the cursor. Collection stops rather than resuming from
diff --git a/internal/storage/lock.go b/internal/storage/lock.go
index 30ca6d7..f8e984f 100644
--- a/internal/storage/lock.go
+++ b/internal/storage/lock.go
@@ -99,6 +99,48 @@ func LockExportWait(root string, timeout time.Duration) (*SessionLock, error) {
}
}
+// ScenarioDir is the directory a claude-code scenario build creates in its
+// storage root. A pipeline over a root that has it takes the scenario lock
+// there, because a pipeline over such a root removes sessions.
+//
+// The name starts with an underscore, so a storage root's session listing
+// passes over it as it passes over _conversations.
+const ScenarioDir = "_scenario"
+
+// ErrScenarioBusy means another pipeline holds this scenario root.
+var ErrScenarioBusy = errors.New("storage: the scenario root is held by another pipeline")
+
+// LockScenario takes the lock one pipeline holds over a scenario root for as
+// long as it runs.
+//
+// A pipeline over a scenario root removes a session once all of it is sent.
+// A second pipeline over the same root could land a removed session again
+// through a source it listed a moment earlier, or parse evidence that is
+// gone. So only one pipeline works on such a root at a time. A rolling
+// update that starts a new process before the old one stops makes two of
+// them an ordinary case.
+func LockScenario(root string) (*SessionLock, error) {
+ l, err := lockDir(filepath.Join(root, ScenarioDir))
+ if errors.Is(err, ErrSessionBusy) {
+ return nil, ErrScenarioBusy
+ }
+ return l, err
+}
+
+// LockScenarioWait is LockScenario, waiting up to timeout for the lock
+// rather than giving up at once. A single pass waits, because it has to do
+// its work or say it could not. A watching pipeline tries on every pass.
+func LockScenarioWait(root string, timeout time.Duration) (*SessionLock, error) {
+ deadline := time.Now().Add(timeout)
+ for {
+ l, err := LockScenario(root)
+ if !errors.Is(err, ErrScenarioBusy) || time.Now().After(deadline) {
+ return l, err
+ }
+ time.Sleep(100 * time.Millisecond)
+ }
+}
+
// Unlock releases the lock.
func (l *SessionLock) Unlock() error {
if l == nil || l.f == nil {
diff --git a/internal/storage/lock_test.go b/internal/storage/lock_test.go
new file mode 100644
index 0000000..13bd07b
--- /dev/null
+++ b/internal/storage/lock_test.go
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package storage_test
+
+import (
+ "errors"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// TestLockScenarioReportsBusy. One pipeline works on a scenario root at a
+// time. A second one must be told the root is held rather than take it too,
+// and a single pass that waits for it must give up at its deadline.
+func TestLockScenarioReportsBusy(t *testing.T) {
+ root := t.TempDir()
+ held, err := storage.LockScenario(root)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if _, err := os.Stat(filepath.Join(root, storage.ScenarioDir, ".lock")); err != nil {
+ t.Fatalf("the lock is not under %s: %v", storage.ScenarioDir, err)
+ }
+ if _, err := storage.LockScenario(root); !errors.Is(err, storage.ErrScenarioBusy) {
+ t.Fatalf("a second LockScenario gave %v, want ErrScenarioBusy", err)
+ }
+ start := time.Now()
+ if _, err := storage.LockScenarioWait(root, 200*time.Millisecond); !errors.Is(err, storage.ErrScenarioBusy) {
+ t.Fatalf("LockScenarioWait gave %v, want ErrScenarioBusy", err)
+ }
+ if waited := time.Since(start); waited < 200*time.Millisecond {
+ t.Fatalf("LockScenarioWait gave up after %s, before its deadline", waited)
+ }
+ if err := held.Unlock(); err != nil {
+ t.Fatal(err)
+ }
+ again, err := storage.LockScenarioWait(root, time.Second)
+ if err != nil {
+ t.Fatalf("the lock was not free once released: %v", err)
+ }
+ if err := again.Unlock(); err != nil {
+ t.Fatal(err)
+ }
+}
diff --git a/internal/storage/removed.go b/internal/storage/removed.go
new file mode 100644
index 0000000..6546fcd
--- /dev/null
+++ b/internal/storage/removed.go
@@ -0,0 +1,273 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package storage
+
+import (
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+)
+
+// RemovedDir is the directory under a storage root where a removal moves a
+// directory before it deletes it.
+//
+// Deleting a session directory in place takes one call per file, and a walk
+// in name order reaches index/ before streams/. A crash in the middle left a
+// session with no index and some of its landed files. A parse then rebuilt
+// the index from what was left and published a round of tombstones. A rename
+// is one call. Before it the directory is whole, and after it no listing sees
+// any of it, because every listing of a root passes over names that start
+// with an underscore.
+const RemovedDir = "_removed"
+
+// RemoveDir removes dir whole: one rename to <root>/_removed/<name>, then a
+// delete there with DeleteTree.
+//
+// A dir that does not exist is not an error. A removal that stopped after its
+// rename meets it that way when it runs again, and then deletes what it moved
+// under the same name. dir must be a directory under root, and not a symbolic
+// link, so a link can never lead a removal out of the root. For the same
+// reason <root>/_removed must be a real directory, or absent, and anything
+// else there is refused with ErrUnsafeRemoved. A target that exists already
+// is refused. Only a leftover that could not be deleted leaves one, and a
+// person has to look at it.
+//
+// The caller releases every lock held inside dir first. Windows cannot rename
+// a directory that holds a file opened with no sharing, and that is how a
+// lock is held there.
+//
+// hook, when it is not nil, is called with "renamed" after the rename, and
+// with "deleting" after the first entry of the moved tree is deleted. An
+// error from it stops RemoveDir there, as a crash would. The tests use it.
+func RemoveDir(root, dir, name string, hook func(where string) error) error {
+ if name == "" || name == "." || name == ".." || strings.ContainsAny(name, `/\`) {
+ return fmt.Errorf("storage: %q is not a name for an entry of %s", name, RemovedDir)
+ }
+ rel, err := filepath.Rel(root, dir)
+ if err != nil || rel == "." || !filepath.IsLocal(rel) ||
+ rel == RemovedDir || strings.HasPrefix(rel, RemovedDir+string(filepath.Separator)) {
+ return fmt.Errorf("storage: %s is not a directory under %s that a removal may take", dir, root)
+ }
+ removed, exists, err := removedDir(root)
+ if err != nil {
+ return err
+ }
+ target := filepath.Join(removed, name)
+ var deleting func() error
+ if hook != nil {
+ deleting = func() error { return hook("deleting") }
+ }
+ fi, err := os.Lstat(dir)
+ switch {
+ case errors.Is(err, fs.ErrNotExist):
+ return DeleteTree(target, deleting)
+ case err != nil:
+ return err
+ case !fi.IsDir():
+ return fmt.Errorf("storage: %s is a symbolic link or not a directory, so it is not removed", dir)
+ }
+ if _, err := os.Lstat(target); err == nil {
+ return fmt.Errorf("%w: %s is left from an earlier removal, so %s is not removed", ErrExists, target, dir)
+ } else if !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ if !exists {
+ // Mkdir, never MkdirAll, and the check once more after it. A link
+ // someone made at _removed since the first check makes Mkdir fail
+ // as existing, and the check then refuses it.
+ if err := os.Mkdir(removed, 0o755); err != nil && !errors.Is(err, fs.ErrExist) {
+ return err
+ }
+ if _, _, err := removedDir(root); err != nil {
+ return err
+ }
+ }
+ if err := os.Rename(dir, target); err != nil {
+ return err
+ }
+ // Either side of the rename is a whole directory, so a crash that loses
+ // the rename is safe. Syncing both directories only narrows that window.
+ syncDir(filepath.Dir(dir))
+ syncDir(filepath.Dir(target))
+ if hook != nil {
+ if err := hook("renamed"); err != nil {
+ return err
+ }
+ }
+ return DeleteTree(target, deleting)
+}
+
+// DeleteTree deletes dir and everything under it: every file first, then the
+// directories, deepest first.
+//
+// A landed file is read-only by contract. Each file is made writable before
+// it is deleted, as the scenario runner does, so the delete does not depend
+// on how a platform treats a read-only file. A directory is made readable
+// and writable before its entries are read. A symbolic link is deleted as a
+// link. Nothing it points at is read, changed or deleted. A dir that does not
+// exist is not an error.
+//
+// after, when it is not nil, is called once, right after the first entry is
+// deleted. An error from it stops DeleteTree there, as a crash would.
+func DeleteTree(dir string, after func() error) error {
+ fi, err := os.Lstat(dir)
+ if err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ return err
+ }
+ called := false
+ deleted := func() error {
+ if called || after == nil {
+ return nil
+ }
+ called = true
+ return after()
+ }
+ if !fi.IsDir() {
+ if err := removeEntry(dir, fi.Mode().IsRegular()); err != nil {
+ return err
+ }
+ return deleted()
+ }
+ type entry struct {
+ path string
+ regular bool
+ }
+ var files []entry
+ var dirs []string
+ err = filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ if !d.IsDir() {
+ files = append(files, entry{p, d.Type().IsRegular()})
+ return nil
+ }
+ // WalkDir calls this before it reads the directory, so a directory
+ // someone made read-only can still be listed and emptied.
+ if info, ierr := d.Info(); ierr == nil && info.Mode().Perm()&0o700 != 0o700 {
+ if err := os.Chmod(p, info.Mode().Perm()|0o700); err != nil {
+ return err
+ }
+ }
+ dirs = append(dirs, p)
+ return nil
+ })
+ if err != nil {
+ return err
+ }
+ for _, f := range files {
+ if err := removeEntry(f.path, f.regular); err != nil {
+ return err
+ }
+ if err := deleted(); err != nil {
+ return err
+ }
+ }
+ // WalkDir lists a directory before everything under it, so the reverse
+ // order deletes every directory after its children.
+ for i := len(dirs) - 1; i >= 0; i-- {
+ if err := removeEntry(dirs[i], false); err != nil {
+ return err
+ }
+ if err := deleted(); err != nil {
+ return err
+ }
+ }
+ return nil
+}
+
+// ErrUnsafeRemoved says <root>/_removed exists and is a symbolic link or not
+// a directory. Nothing is read, deleted or moved through it.
+var ErrUnsafeRemoved = errors.New("is a symbolic link or not a directory, so nothing is removed")
+
+// removedDir checks <root>/_removed before a removal reads it or moves
+// anything into it, and reports whether it exists. It uses Lstat, which
+// does not follow a link. A link there would lead the sweep to delete what
+// the link points at, and a rename to move a session directory out of the
+// root. Before this check, a _removed linked to a directory holding a
+// transcript and a read-only file left that directory empty, with no error.
+func removedDir(root string) (dir string, exists bool, err error) {
+ dir = filepath.Join(root, RemovedDir)
+ fi, err := os.Lstat(dir)
+ switch {
+ case errors.Is(err, fs.ErrNotExist):
+ return dir, false, nil
+ case err != nil:
+ return dir, false, err
+ case fi.Mode().Type() != fs.ModeDir:
+ return dir, false, fmt.Errorf("storage: %s %w", dir, ErrUnsafeRemoved)
+ }
+ return dir, true, nil
+}
+
+// SweepRemoved deletes every entry under <root>/_removed. Each is what a
+// removal moved there and did not finish deleting, because the process
+// stopped or a delete failed. Every entry is tried, and the failures are
+// returned together. A root with no _removed is left as it is. A _removed
+// that is a symbolic link, or not a directory, is refused with
+// ErrUnsafeRemoved, and nothing is read or deleted.
+func SweepRemoved(root string) error {
+ dir, exists, err := removedDir(root)
+ if err != nil || !exists {
+ return err
+ }
+ items, err := os.ReadDir(dir)
+ if err != nil {
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ return err
+ }
+ var errs []error
+ for _, it := range items {
+ if err := DeleteTree(filepath.Join(dir, it.Name()), nil); err != nil {
+ errs = append(errs, err)
+ }
+ }
+ return errors.Join(errs...)
+}
+
+// removeEntry deletes one entry. An entry that is already gone counts as
+// deleted, so a delete that stopped halfway can simply run again.
+func removeEntry(path string, regular bool) error {
+ if regular {
+ // A failure here is left to the delete, which reports it.
+ _ = os.Chmod(path, 0o644)
+ }
+ if err := os.Remove(path); err != nil && !errors.Is(err, fs.ErrNotExist) {
+ return err
+ }
+ return nil
+}
+
+// syncDir flushes a directory's entries to disk, as far as the platform
+// allows. It is best effort by design.
+func syncDir(dir string) {
+ d, err := os.Open(dir)
+ if err != nil {
+ return
+ }
+ _ = d.Sync()
+ _ = d.Close()
+}
diff --git a/internal/storage/removed_test.go b/internal/storage/removed_test.go
new file mode 100644
index 0000000..484fee1
--- /dev/null
+++ b/internal/storage/removed_test.go
@@ -0,0 +1,410 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package storage_test
+
+import (
+ "errors"
+ "io"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+)
+
+// sessionTree writes the shape a session directory has: a lock, an index,
+// read-only landed files, and cursors. It returns how many files it wrote.
+func sessionTree(t *testing.T, dir string) int {
+ t.Helper()
+ write := func(p string, perm os.FileMode) {
+ if err := storage.WriteAtomic(p, perm, func(w io.Writer) error {
+ _, err := io.WriteString(w, "{}\n")
+ return err
+ }); err != nil {
+ t.Fatal(err)
+ }
+ }
+ write(filepath.Join(dir, ".lock"), storage.PermState)
+ write(filepath.Join(dir, "index", "index.state"), storage.PermState)
+ write(filepath.Join(dir, "index", "entries.bin"), storage.PermLanded)
+ write(filepath.Join(dir, "streams", "main", storage.LandedName("transcript", storage.Stamp(time.Unix(0, 0)), 1)), storage.PermLanded)
+ write(filepath.Join(dir, "streams", "main", "transcript.cursor"), storage.PermState)
+ write(filepath.Join(dir, "runs", "wf_r1", storage.LandedName("journal", storage.Stamp(time.Unix(0, 0)), 2)), storage.PermLanded)
+ return 6
+}
+
+// countFiles counts the entries under dir that are not directories, and
+// gives 0 when dir is gone.
+func countFiles(t *testing.T, dir string) int {
+ t.Helper()
+ if _, err := os.Lstat(dir); errors.Is(err, fs.ErrNotExist) {
+ return 0
+ }
+ n := 0
+ err := filepath.WalkDir(dir, func(_ string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ if !d.IsDir() {
+ n++
+ }
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return n
+}
+
+func gone(t *testing.T, path string) bool {
+ t.Helper()
+ _, err := os.Lstat(path)
+ return errors.Is(err, fs.ErrNotExist)
+}
+
+// A removal deletes landed files, which are read-only by contract, and a
+// directory someone made read-only. It says so once, after the first entry,
+// which is where the crash tests stop it.
+func TestDeleteTreeDeletesReadOnlyFilesAndCallsAfterOnce(t *testing.T) {
+ dir := filepath.Join(t.TempDir(), "S")
+ total := sessionTree(t, dir)
+ locked := filepath.Join(dir, "runs", "wf_r1")
+ if err := os.Chmod(locked, 0o555); err != nil {
+ t.Fatal(err)
+ }
+ // If the test fails before the delete, the temporary directory must
+ // still be removable.
+ t.Cleanup(func() { _ = os.Chmod(locked, 0o755) })
+
+ calls, left := 0, -1
+ err := storage.DeleteTree(dir, func() error {
+ calls++
+ left = countFiles(t, dir)
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if calls != 1 || left != total-1 {
+ t.Fatalf("after ran %d times with %d files left; want once, with %d left", calls, left, total-1)
+ }
+ if !gone(t, dir) {
+ t.Fatalf("%s is still there", dir)
+ }
+ if err := storage.DeleteTree(dir, nil); err != nil {
+ t.Fatalf("a tree that is gone already is not an error: %v", err)
+ }
+}
+
+// A symbolic link inside the tree is deleted as a link. What it points at,
+// outside the tree, is not touched, and its mode is not changed.
+func TestDeleteTreeNeverFollowsALink(t *testing.T) {
+ base := t.TempDir()
+ outside := filepath.Join(base, "outside")
+ kept := filepath.Join(outside, "real.sd")
+ if err := storage.WriteAtomic(kept, storage.PermLanded, func(w io.Writer) error {
+ _, err := io.WriteString(w, "keep me\n")
+ return err
+ }); err != nil {
+ t.Fatal(err)
+ }
+ dir := filepath.Join(base, "S")
+ sessionTree(t, dir)
+ if err := os.Symlink(outside, filepath.Join(dir, "streams", "to-a-directory")); err != nil {
+ t.Skipf("this system makes no symbolic links: %v", err)
+ }
+ if err := os.Symlink(kept, filepath.Join(dir, "to-a-file")); err != nil {
+ t.Skipf("this system makes no symbolic links: %v", err)
+ }
+ if err := storage.DeleteTree(dir, nil); err != nil {
+ t.Fatal(err)
+ }
+ if !gone(t, dir) {
+ t.Fatalf("%s is still there", dir)
+ }
+ fi, err := os.Stat(kept)
+ if err != nil {
+ t.Fatalf("the file a link pointed at was deleted: %v", err)
+ }
+ if fi.Mode().Perm() != storage.PermLanded {
+ t.Fatalf("the file a link pointed at changed mode to %v", fi.Mode().Perm())
+ }
+}
+
+// An error from after stops the delete where it is, as a crash would. The
+// rest of the tree stays for the next attempt, which finishes it.
+func TestDeleteTreeStopsWhereAfterFails(t *testing.T) {
+ dir := filepath.Join(t.TempDir(), "S")
+ total := sessionTree(t, dir)
+ stop := errors.New("stop")
+ if err := storage.DeleteTree(dir, func() error { return stop }); !errors.Is(err, stop) {
+ t.Fatalf("DeleteTree gave %v, want the error after returned", err)
+ }
+ if n := countFiles(t, dir); n != total-1 {
+ t.Fatalf("%d files left, want %d: the delete must stop right after the first", n, total-1)
+ }
+ if err := storage.DeleteTree(dir, nil); err != nil {
+ t.Fatal(err)
+ }
+ if !gone(t, dir) {
+ t.Fatalf("%s is still there", dir)
+ }
+}
+
+// A removal renames the directory away in one call, so no reader ever sees
+// half of it, and then deletes it where no listing looks.
+func TestRemoveDirRenamesInOneCallThenDeletes(t *testing.T) {
+ root := t.TempDir()
+ dir := filepath.Join(root, "S")
+ total := sessionTree(t, dir)
+ moved := filepath.Join(root, storage.RemovedDir, "S")
+ var seen []string
+ err := storage.RemoveDir(root, dir, "S", func(where string) error {
+ seen = append(seen, where)
+ switch where {
+ case "renamed":
+ if !gone(t, dir) {
+ t.Errorf("after the rename %s is still there", dir)
+ }
+ if n := countFiles(t, moved); n != total {
+ t.Errorf("after the rename %d of %d files are under %s", n, total, moved)
+ }
+ case "deleting":
+ if n := countFiles(t, moved); n != total-1 {
+ t.Errorf("after the first delete %d files are left, want %d", n, total-1)
+ }
+ }
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if strings.Join(seen, ",") != "renamed,deleting" {
+ t.Fatalf("the hook saw %v, want renamed then deleting", seen)
+ }
+ if !gone(t, moved) {
+ t.Fatalf("%s is still there", moved)
+ }
+ items, err := os.ReadDir(filepath.Join(root, storage.RemovedDir))
+ if err != nil || len(items) != 0 {
+ t.Fatalf("%s holds %d entries, want none: %v", storage.RemovedDir, len(items), err)
+ }
+}
+
+// A removal stopped after its rename leaves the whole directory under
+// _removed. The next attempt deletes it: RemoveDir again, which finds dir
+// gone, or the sweep a removal starts with.
+func TestARemovalStoppedAfterTheRenameIsFinishedLater(t *testing.T) {
+ stop := errors.New("stop")
+ for _, finish := range []struct {
+ name string
+ fn func(root, dir string) error
+ }{
+ {"again", func(root, dir string) error { return storage.RemoveDir(root, dir, "S", nil) }},
+ {"sweep", func(root, _ string) error { return storage.SweepRemoved(root) }},
+ } {
+ t.Run(finish.name, func(t *testing.T) {
+ root := t.TempDir()
+ dir := filepath.Join(root, "S")
+ total := sessionTree(t, dir)
+ moved := filepath.Join(root, storage.RemovedDir, "S")
+ err := storage.RemoveDir(root, dir, "S", func(where string) error {
+ if where == "renamed" {
+ return stop
+ }
+ return nil
+ })
+ if !errors.Is(err, stop) {
+ t.Fatalf("RemoveDir gave %v, want the hook's error", err)
+ }
+ if n := countFiles(t, moved); n != total || !gone(t, dir) {
+ t.Fatalf("after a stop at the rename: %d of %d files moved, and %s gone: %v", n, total, dir, gone(t, dir))
+ }
+ if err := finish.fn(root, dir); err != nil {
+ t.Fatal(err)
+ }
+ if !gone(t, moved) {
+ t.Fatalf("%s is still there", moved)
+ }
+ })
+ }
+}
+
+// A removal never takes a path outside the root, never follows a link, and
+// never overwrites a leftover a person has to look at.
+func TestRemoveDirRefusesWhatItMustNotTake(t *testing.T) {
+ root := t.TempDir()
+ elsewhere := t.TempDir()
+ total := sessionTree(t, filepath.Join(elsewhere, "S"))
+
+ if err := storage.RemoveDir(root, filepath.Join(elsewhere, "S"), "S", nil); err == nil {
+ t.Fatal("a directory outside the root was taken")
+ }
+ if err := storage.RemoveDir(root, root, "S", nil); err == nil {
+ t.Fatal("the root itself was taken")
+ }
+ if err := storage.RemoveDir(root, filepath.Join(root, storage.RemovedDir), "x", nil); err == nil {
+ t.Fatalf("%s itself was taken", storage.RemovedDir)
+ }
+ for _, name := range []string{"", ".", "..", "a/b", `a\b`} {
+ if err := storage.RemoveDir(root, filepath.Join(root, "S"), name, nil); err == nil {
+ t.Fatalf("the name %q was accepted", name)
+ }
+ }
+
+ // A file is not a directory, and it is left as it was.
+ file := filepath.Join(root, "F")
+ if err := os.WriteFile(file, []byte("x"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := storage.RemoveDir(root, file, "F", nil); err == nil {
+ t.Fatal("a file was taken as a directory")
+ }
+ if gone(t, file) {
+ t.Fatal("the refused file was deleted")
+ }
+
+ // A leftover under the same name stops the removal, and the directory
+ // stays whole.
+ dir := filepath.Join(root, "S2")
+ n := sessionTree(t, dir)
+ if err := os.MkdirAll(filepath.Join(root, storage.RemovedDir, "S2"), 0o755); err != nil {
+ t.Fatal(err)
+ }
+ if err := storage.RemoveDir(root, dir, "S2", nil); !errors.Is(err, storage.ErrExists) {
+ t.Fatalf("RemoveDir over a leftover gave %v, want ErrExists", err)
+ }
+ if got := countFiles(t, dir); got != n {
+ t.Fatalf("the refused directory holds %d of its %d files", got, n)
+ }
+
+ // A link is refused, and nothing behind it is touched.
+ link := filepath.Join(root, "S3")
+ if err := os.Symlink(filepath.Join(elsewhere, "S"), link); err != nil {
+ t.Skipf("this system makes no symbolic links: %v", err)
+ }
+ if err := storage.RemoveDir(root, link, "S3", nil); err == nil {
+ t.Fatal("a symbolic link was taken as a directory")
+ }
+ if gone(t, link) {
+ t.Fatal("the refused link was deleted")
+ }
+ if got := countFiles(t, filepath.Join(elsewhere, "S")); got != total {
+ t.Fatalf("the directory behind the link holds %d of its %d files", got, total)
+ }
+}
+
+// A _removed that is a symbolic link would lead the sweep to delete what it
+// points at, and a rename to move a session directory out of the root.
+// Before the check, a link to a directory holding a transcript and a
+// read-only file left that directory empty, with no error. Both refuse it
+// now, and every file behind the link survives with its mode.
+func TestALinkAtRemovedIsRefused(t *testing.T) {
+ base := t.TempDir()
+ outside := filepath.Join(base, "outside")
+ precious := filepath.Join(outside, "precious.jsonl")
+ readOnly := filepath.Join(outside, "keepdir", "x")
+ // Named as a session a stopped removal would finish, so RemoveDir of a
+ // directory that is gone reaches it through the link.
+ leftover := filepath.Join(outside, "S2", "streams", "main", "kept.sd")
+ for _, p := range []string{precious, readOnly, leftover} {
+ if err := storage.WriteAtomic(p, storage.PermLanded, func(w io.Writer) error {
+ _, err := io.WriteString(w, "keep me\n")
+ return err
+ }); err != nil {
+ t.Fatal(err)
+ }
+ }
+ root := filepath.Join(base, "root")
+ dir := filepath.Join(root, "S")
+ total := sessionTree(t, dir)
+ if err := os.Symlink(outside, filepath.Join(root, storage.RemovedDir)); err != nil {
+ t.Skipf("this system makes no symbolic links: %v", err)
+ }
+
+ if err := storage.SweepRemoved(root); !errors.Is(err, storage.ErrUnsafeRemoved) {
+ t.Fatalf("the sweep through a link gave %v, want ErrUnsafeRemoved", err)
+ }
+ if err := storage.RemoveDir(root, dir, "S", nil); !errors.Is(err, storage.ErrUnsafeRemoved) {
+ t.Fatalf("RemoveDir through a link gave %v, want ErrUnsafeRemoved", err)
+ }
+ if got := countFiles(t, dir); got != total {
+ t.Fatalf("the refused directory holds %d of its %d files", got, total)
+ }
+ if err := storage.RemoveDir(root, filepath.Join(root, "S2"), "S2", nil); !errors.Is(err, storage.ErrUnsafeRemoved) {
+ t.Fatalf("RemoveDir of a directory that is gone, through a link, gave %v, want ErrUnsafeRemoved", err)
+ }
+ for _, p := range []string{precious, readOnly, leftover} {
+ fi, err := os.Stat(p)
+ if err != nil {
+ t.Fatalf("%s, behind the link, was deleted: %v", p, err)
+ }
+ if fi.Mode().Perm() != storage.PermLanded {
+ t.Fatalf("%s, behind the link, changed mode to %v", p, fi.Mode().Perm())
+ }
+ }
+ if !gone(t, filepath.Join(outside, "S")) {
+ t.Fatal("a session directory was moved through the link")
+ }
+}
+
+// A _removed that is a plain file is refused the same way, and it stays.
+func TestAFileAtRemovedIsRefused(t *testing.T) {
+ root := t.TempDir()
+ dir := filepath.Join(root, "S")
+ total := sessionTree(t, dir)
+ file := filepath.Join(root, storage.RemovedDir)
+ if err := os.WriteFile(file, []byte("not a directory\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if err := storage.SweepRemoved(root); !errors.Is(err, storage.ErrUnsafeRemoved) {
+ t.Fatalf("the sweep over a file gave %v, want ErrUnsafeRemoved", err)
+ }
+ if err := storage.RemoveDir(root, dir, "S", nil); !errors.Is(err, storage.ErrUnsafeRemoved) {
+ t.Fatalf("RemoveDir into a file gave %v, want ErrUnsafeRemoved", err)
+ }
+ if got := countFiles(t, dir); got != total {
+ t.Fatalf("the refused directory holds %d of its %d files", got, total)
+ }
+ if data, err := os.ReadFile(file); err != nil || string(data) != "not a directory\n" {
+ t.Fatalf("the file at %s changed: %q, %v", storage.RemovedDir, data, err)
+ }
+}
+
+// The sweep deletes every leftover, and a root with none is left as it is.
+func TestSweepRemovedDeletesEveryLeftover(t *testing.T) {
+ root := t.TempDir()
+ if err := storage.SweepRemoved(root); err != nil {
+ t.Fatal(err)
+ }
+ if !gone(t, filepath.Join(root, storage.RemovedDir)) {
+ t.Fatalf("the sweep made %s in a root that had none", storage.RemovedDir)
+ }
+ sessionTree(t, filepath.Join(root, storage.RemovedDir, "S"))
+ sessionTree(t, filepath.Join(root, storage.RemovedDir, "S.chain"))
+ if err := storage.SweepRemoved(root); err != nil {
+ t.Fatal(err)
+ }
+ items, err := os.ReadDir(filepath.Join(root, storage.RemovedDir))
+ if err != nil || len(items) != 0 {
+ t.Fatalf("%s holds %d entries after the sweep, want none: %v", storage.RemovedDir, len(items), err)
+ }
+}
diff --git a/internal/storage/zone.go b/internal/storage/zone.go
index 4a66049..d092216 100644
--- a/internal/storage/zone.go
+++ b/internal/storage/zone.go
@@ -50,6 +50,11 @@ func (z *Zone) Root() string { return z.root }
// SessionDir is the directory holding everything collected for one session.
// It is the unit of retention: purging a session is removing this directory.
+//
+// asz never removes it for a real session, so what was landed outlives the
+// source. The one exception is a session a scenario build marked. Once all
+// of it is sent, a pipeline renames this directory into RemovedDir and
+// deletes it there, so no reader ever sees half of it.
func (z *Zone) SessionDir(session string) string {
return filepath.Join(z.root, session)
}
diff --git a/tests/chain/remove_test.go b/tests/chain/remove_test.go
new file mode 100644
index 0000000..ea01a69
--- /dev/null
+++ b/tests/chain/remove_test.go
@@ -0,0 +1,602 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package chain_test
+
+import (
+ "errors"
+ "fmt"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecode"
+ "github.com/apache/skywalking-ai-sessionizer/internal/adapters/claudecodechanges"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp"
+ "github.com/apache/skywalking-ai-sessionizer/internal/export/otlp/otlptest"
+ "github.com/apache/skywalking-ai-sessionizer/internal/metrics"
+ "github.com/apache/skywalking-ai-sessionizer/internal/parse"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario"
+ "github.com/apache/skywalking-ai-sessionizer/internal/scenario/remove"
+ "github.com/apache/skywalking-ai-sessionizer/internal/storage"
+ "github.com/apache/skywalking-ai-sessionizer/internal/view"
+)
+
+// What a scenario cannot express about removing a sent scenario session: a
+// removal stopped at every point it reaches, as a crash stops it, the locks
+// it waits for, and a build that has not finished. Each test drives the
+// real adapters, deriver, parser, pusher and remover over the
+// removed-after-sent scenario.
+
+// removalBudget is the round budget removed-after-sent's expectation file
+// sets. The chain then holds several rounds, so a round parsed or sent again
+// would show.
+const removalBudget = 3072
+
+// removalScenario is removed-after-sent: one session with every kind of
+// file a removal deletes.
+func removalScenario(t *testing.T) *scenario.Scenario {
+ t.Helper()
+ sc, err := scenario.Load(filepath.Join("..", "scenarios", "removed-after-sent.yaml"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ return sc
+}
+
+// sentRoot is a scenario root and the receiver its pipeline sends to.
+type sentRoot struct {
+ t *testing.T
+ dir string
+ source string
+ zone *storage.Zone
+ rcv *otlptest.Receiver
+ endpoint string
+}
+
+func newSentRoot(t *testing.T) *sentRoot {
+ t.Helper()
+ // Claude Code's own directory is a place of the test's, so the guard on
+ // it compares against something known on every machine.
+ t.Setenv("CLAUDE_CONFIG_DIR", t.TempDir())
+ rcv, err := otlptest.Start()
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(rcv.Close)
+ o := rcv.Options(otlp.ProtocolGRPC)
+ return rootAt(t, t.TempDir(), rcv, otlp.EndpointOf(o.Protocol, o.Endpoint, o.TLS))
+}
+
+func rootAt(t *testing.T, dir string, rcv *otlptest.Receiver, endpoint string) *sentRoot {
+ return &sentRoot{t: t, dir: dir, source: filepath.Join(dir, "_source"), zone: storage.NewZone(dir), rcv: rcv, endpoint: endpoint}
+}
+
+func (r *sentRoot) build(sc *scenario.Scenario, opts scenario.Options) *scenario.Built {
+ r.t.Helper()
+ opts.At = time.Date(2026, 2, 1, 10, 0, 0, 0, time.UTC)
+ b, err := scenario.Build(sc, scenario.FormatClaudeCode, r.dir, opts)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ return b
+}
+
+func (r *sentRoot) changesRoot() string { return filepath.Join(r.source, "plugins", "data") }
+
+// collect lands with new collectors for both adapters, as a process that has
+// just started does, and returns how many sources it landed from.
+func (r *sentRoot) collect() int {
+ r.t.Helper()
+ st, err := claudecode.New(r.source, r.zone, 0).CollectAll(nil)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ cs, err := claudecodechanges.New(r.changesRoot(), r.zone, 0).CollectAll(nil)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ if len(st.Errors) > 0 || len(cs.Errors) > 0 {
+ r.t.Fatalf("collect: %v %v", st.Errors, cs.Errors)
+ }
+ return st.SourcesLanded + cs.SourcesLanded
+}
+
+// derive runs a new deriver over what is landed, and returns how many
+// requests it put in the spool.
+func (r *sentRoot) derive() int {
+ r.t.Helper()
+ st, err := (&metrics.Deriver{Zone: r.zone, Options: metrics.Options{Grace: -1, Version: "test"}}).Pass(nil)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ if len(st.Errors) > 0 {
+ r.t.Fatalf("derive: %v", st.Errors)
+ }
+ return st.Requests
+}
+
+// parse writes rounds for every session directory, as a pass of the
+// pipeline does, and returns how many it wrote.
+func (r *sentRoot) parse() int {
+ r.t.Helper()
+ items, err := os.ReadDir(r.dir)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ written := 0
+ for _, it := range items {
+ if !it.IsDir() || strings.HasPrefix(it.Name(), "_") {
+ continue
+ }
+ for {
+ res, err := parse.Session(r.zone, parse.Options{Conversation: it.Name(), Session: it.Name(), MaxRoundBytes: removalBudget})
+ if err != nil {
+ r.t.Fatalf("parse %s: %v", it.Name(), err)
+ }
+ if res.Changed() {
+ written++
+ }
+ if !res.Changed() || !res.More {
+ break
+ }
+ }
+ }
+ return written
+}
+
+// push sends what is not sent yet, and records the receiver in push.state,
+// as the pipeline's pusher does.
+func (r *sentRoot) push() {
+ r.t.Helper()
+ client, err := r.rcv.Client(otlp.ProtocolGRPC)
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ defer client.Close()
+ p := &otlp.Pusher{Zone: r.zone, Client: client, Endpoint: r.endpoint, Version: "test",
+ ServiceName: "Removal Test", InstanceID: "removal-test", Layer: "AI_AGENT"}
+ st, err := p.Pass()
+ if err != nil {
+ r.t.Fatal(err)
+ }
+ if len(st.Errors) > 0 {
+ r.t.Fatalf("push: %v", st.Errors)
+ }
+}
+
+// pipeline is one pass of asz collect before its removal.
+func (r *sentRoot) pipeline() {
+ r.t.Helper()
+ r.collect()
+ r.derive()
+ r.parse()
+ r.push()
+}
+
+func (r *sentRoot) remover() *remove.Remover {
+ return &remove.Remover{Zone: r.zone, Source: r.source, Local: claudecode.New(r.source, r.zone, 0),
+ Changes: claudecodechanges.New(r.changesRoot(), r.zone, 0), ChangesRoot: r.changesRoot(), Derives: true}
+}
+
+func (r *sentRoot) input() remove.Input {
+ return remove.Input{Endpoint: r.endpoint, SendLogs: true, SendMetrics: true}
+}
+
+// requests counts what reached the receiver since it was last reset.
+func (r *sentRoot) requests() int { return len(r.rcv.Requests()) + len(r.rcv.MetricsRequests()) }
+
+// copyOf is a full copy of the root, modes kept, that sends to the same
+// receiver. Nothing in a root records where the root is, so a copy is a root
+// in its own right. It stands for a fresh root that was built, landed,
+// derived, parsed and sent, at a small part of the time that takes.
+func (r *sentRoot) copyOf(t *testing.T) *sentRoot {
+ t.Helper()
+ dir := t.TempDir()
+ err := filepath.WalkDir(r.dir, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ rel, err := filepath.Rel(r.dir, p)
+ if err != nil {
+ return err
+ }
+ info, err := d.Info()
+ if err != nil {
+ return err
+ }
+ to := filepath.Join(dir, rel)
+ if d.IsDir() {
+ return os.MkdirAll(to, info.Mode().Perm())
+ }
+ data, err := os.ReadFile(p)
+ if err != nil {
+ return err
+ }
+ return os.WriteFile(to, data, info.Mode().Perm())
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return rootAt(t, dir, r.rcv, r.endpoint)
+}
+
+// in is what a build of the root this was copied from reports, with its
+// paths moved into this copy.
+func (r *sentRoot) in(b *scenario.Built) *scenario.Built {
+ moved := *b
+ moved.Out, moved.Marker = r.dir, scenario.MarkerPath(r.source, b.Session)
+ return &moved
+}
+
+// gone checks that nothing of the session is left, and that nothing other
+// sessions share went with it.
+func (r *sentRoot) gone(b *scenario.Built, m *scenario.Marker) {
+ r.t.Helper()
+ t, s := r.t, b.Session
+ if present(b.Marker) {
+ t.Errorf("the marker %s is still there", b.Marker)
+ }
+ for _, f := range m.Files {
+ if present(filepath.Join(r.source, filepath.FromSlash(f.Path))) {
+ t.Errorf("the source file %s is still there", f.Path)
+ }
+ if i := strings.Index(f.Path, "/"+s+"/"); i >= 0 {
+ own := filepath.Join(r.source, filepath.FromSlash(f.Path[:i+1+len(s)]))
+ if present(own) {
+ t.Errorf("the directory %s, named for the session, is still there", own)
+ }
+ if !present(filepath.Dir(own)) {
+ t.Errorf("%s, which other sessions share, was removed", filepath.Dir(own))
+ }
+ }
+ }
+ for _, p := range []string{r.zone.SessionDir(s), filepath.Join(r.dir, "_conversations", s)} {
+ if present(p) {
+ t.Errorf("%s is still there", p)
+ }
+ }
+ for _, name := range spoolOf(t, r.dir, s) {
+ t.Errorf("the spool file %s is still there", name)
+ }
+ for _, state := range []string{filepath.Join(r.dir, otlp.StateFile), filepath.Join(r.dir, storage.SpoolDir, metrics.StateFile)} {
+ if data, err := os.ReadFile(state); err == nil && strings.Contains(string(data), s) {
+ t.Errorf("%s still names the session", state)
+ }
+ }
+ if items, err := os.ReadDir(filepath.Join(r.dir, storage.RemovedDir)); err == nil && len(items) > 0 {
+ t.Errorf("%s still holds %d entries", storage.RemovedDir, len(items))
+ }
+ project := filepath.Join(r.source, b.Plan.Project)
+ for _, p := range []string{
+ filepath.Join(project, "not-a-uuid.jsonl"),
+ filepath.Join(project, "memory", "notes.md"),
+ filepath.Join(r.source, scenario.MarkerDir),
+ filepath.Join(r.dir, storage.ScenarioDir),
+ filepath.Join(r.source, filepath.FromSlash(scenario.PluginOutputDir)),
+ } {
+ if !present(p) {
+ t.Errorf("%s, which every session shares, was removed", p)
+ }
+ }
+ ids, err := view.New(r.zone, nil).List()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if slices.Contains(ids, s) {
+ t.Error("the page still lists the conversation")
+ }
+}
+
+// evidence lists every landed file, round and spool file a reader of the
+// root can see. What sits under _removed is being deleted and no listing
+// reads it, so a file a rename moved there is not new evidence.
+func evidence(t *testing.T, dir string) map[string]bool {
+ t.Helper()
+ out := map[string]bool{}
+ err := filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
+ if err != nil {
+ return err
+ }
+ if d.IsDir() {
+ if d.Name() == "_source" || d.Name() == storage.RemovedDir {
+ return filepath.SkipDir
+ }
+ return nil
+ }
+ if ext := filepath.Ext(p); ext == ".sd" || ext == ".sf" || ext == ".pb" {
+ rel, _ := filepath.Rel(dir, p)
+ out[filepath.ToSlash(rel)] = true
+ }
+ return nil
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return out
+}
+
+// spoolOf lists the spool files a session owns.
+func spoolOf(t *testing.T, dir, session string) []string {
+ t.Helper()
+ items, err := os.ReadDir(filepath.Join(dir, storage.SpoolDir))
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ var out []string
+ for _, it := range items {
+ if owner, ok := metrics.SpoolOwner(it.Name()); ok && owner == session {
+ out = append(out, it.Name())
+ }
+ }
+ return out
+}
+
+// sentPerFile counts, for every file the receiver holds a record of, how
+// many records named it.
+func sentPerFile(rcv *otlptest.Receiver) map[string]int {
+ out := map[string]int{}
+ for _, rec := range otlptest.Records(rcv.Requests()) {
+ out[otlptest.Attrs(rec.GetAttributes())["asz.file"]]++
+ }
+ return out
+}
+
+// wantPoints is every point a removal of one session reaches, in order: one
+// after each source file of step 1 and each spool file of step 4, and the
+// places inside the rename and delete of steps 2 and 3.
+func wantPoints(session string, files, spool int) []remove.Point {
+ p := []remove.Point{{Step: 0, Where: remove.WhereDone, Session: session}}
+ for k := 1; k <= files; k++ {
+ p = append(p, remove.Point{Step: 1, Where: remove.WhereFile, Session: session, N: k})
+ }
+ p = append(p, remove.Point{Step: 1, Where: remove.WhereDirs, Session: session})
+ for _, step := range []int{2, 3} {
+ for _, where := range []string{remove.WhereUnlocked, remove.WhereRenamed, remove.WhereDeleting, remove.WhereDone} {
+ p = append(p, remove.Point{Step: step, Where: where, Session: session})
+ }
+ }
+ for k := 1; k <= spool; k++ {
+ p = append(p, remove.Point{Step: 4, Where: remove.WhereFile, Session: session, N: k})
+ }
+ return append(p,
+ remove.Point{Step: 4, Where: remove.WhereDone, Session: session},
+ remove.Point{Step: 5, Where: remove.WhereDone},
+ remove.Point{Step: 6, Where: remove.WhereDone},
+ remove.Point{Step: 7, Where: remove.WhereDone, Session: session})
+}
+
+func readMarker(t *testing.T, path string) *scenario.Marker {
+ t.Helper()
+ m, err := scenario.ReadMarker(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return m
+}
+
+func noRemovalErrors(t *testing.T, res *remove.Result) {
+ t.Helper()
+ if len(res.Errors) > 0 {
+ t.Fatalf("the removal reported %v", res.Errors)
+ }
+}
+
+func present(p string) bool {
+ _, err := os.Lstat(p)
+ return err == nil
+}
+
+// TestRemovalFinishesAfterACrash. A removal can stop at any point, as a
+// crash would: between the files of steps 1 and 4, and between the rename
+// and the delete of steps 2 and 3. For every point, the restarted pipeline
+// lands, derives, parses and sends nothing new, and its first pass finishes
+// the removal. That is what the order of the steps is for.
+func TestRemovalFinishesAfterACrash(t *testing.T) {
+ sent := newSentRoot(t)
+ built := sent.build(removalScenario(t), scenario.Options{})
+ m := readMarker(t, built.Marker)
+ sent.pipeline()
+ spool := spoolOf(t, sent.dir, built.Session)
+ if len(spool) == 0 {
+ t.Fatal("the session has no spool file, so step 4 would stop nowhere")
+ }
+
+ // Every point one removal reaches, in order.
+ var points []remove.Point
+ {
+ r := sent.copyOf(t)
+ rm := r.remover()
+ rm.Hook = func(p remove.Point) error { points = append(points, p); return nil }
+ res := rm.Pass(r.input())
+ noRemovalErrors(t, res)
+ if len(res.Removed) != 1 {
+ t.Fatalf("a removal that nothing stopped removed %v, kept %v, waiting %v", res.Removed, res.Kept, res.Waiting)
+ }
+ }
+ if want := wantPoints(built.Session, len(m.Files), len(spool)); !slices.Equal(points, want) {
+ t.Fatalf("a removal reached the points\n%v\nwant\n%v", points, want)
+ }
+
+ errCrash := errors.New("the process stopped here")
+ for i, pt := range points {
+ name := fmt.Sprintf("%02d-step%d-%s", i, pt.Step, pt.Where)
+ if pt.N > 0 {
+ name += fmt.Sprintf("-%d", pt.N)
+ }
+ t.Run(name, func(t *testing.T) {
+ r := sent.copyOf(t)
+ b := r.in(built)
+ before := evidence(t, r.dir)
+ rm := r.remover()
+ reached := 0
+ rm.Hook = func(remove.Point) error {
+ if reached++; reached == i+1 {
+ return errCrash
+ }
+ return nil
+ }
+ if res := rm.Pass(r.input()); len(res.Errors) != 1 || !errors.Is(res.Errors[0], errCrash) {
+ t.Fatalf("the stopped removal reported %v", res.Errors)
+ }
+
+ // The restart: new collectors, a new deriver, a parse of every
+ // session directory, a push and a new remover. The receiver
+ // forgets what it holds, and stands for a fresh one. A new
+ // receiver would listen on another port, which is another
+ // endpoint, and push.state ties what was sent to its endpoint.
+ r.rcv.Reset()
+ if n := r.collect(); n != 0 {
+ t.Errorf("the restart landed %d source(s)", n)
+ }
+ if n := r.derive(); n != 0 {
+ t.Errorf("the restart derived %d request(s)", n)
+ }
+ if n := r.parse(); n != 0 {
+ t.Errorf("the restart wrote %d round(s)", n)
+ }
+ r.push()
+ if n := r.requests(); n != 0 {
+ t.Errorf("the restart sent %d request(s)", n)
+ }
+ for p := range evidence(t, r.dir) {
+ if !before[p] {
+ t.Errorf("the restart made %s, which was not there before", p)
+ }
+ }
+ res := r.remover().Pass(r.input())
+ noRemovalErrors(t, res)
+ // Once step 7 has deleted the marker, nothing is left to finish.
+ if finishes := pt.Step < 7; finishes != slices.Contains(res.Removed, b.Session) {
+ t.Errorf("the restart removed %v", res.Removed)
+ }
+ r.gone(b, m)
+ })
+ }
+}
+
+// TestRemovalWaitsForTheLocks. Whoever holds the session, its chain or the
+// export state is working on it right now. The removal waits and says
+// nothing, since the lock is released on its own. The next pass after the
+// release removes the session.
+func TestRemovalWaitsForTheLocks(t *testing.T) {
+ sent := newSentRoot(t)
+ built := sent.build(removalScenario(t), scenario.Options{})
+ m := readMarker(t, built.Marker)
+ sent.pipeline()
+ for _, which := range []string{"session", "chain", "export"} {
+ t.Run(which, func(t *testing.T) {
+ r := sent.copyOf(t)
+ b := r.in(built)
+ var held *storage.SessionLock
+ var err error
+ switch which {
+ case "session":
+ held, err = storage.LockSession(r.zone.SessionDir(b.Session))
+ case "chain":
+ held, err = storage.LockChain(filepath.Join(r.dir, "_conversations", b.Session))
+ case "export":
+ held, err = storage.LockExport(r.dir)
+ }
+ if err != nil {
+ t.Fatal(err)
+ }
+ rm := r.remover()
+ res := rm.Pass(r.input())
+ noRemovalErrors(t, res)
+ if len(res.Removed) != 0 || len(res.Kept) != 0 || len(res.Waiting) == 0 {
+ t.Fatalf("while the %s lock was held: removed %v, kept %v, waiting %v; the removal must wait and say nothing", which, res.Removed, res.Kept, res.Waiting)
+ }
+ if again := readMarker(t, b.Marker); again.State != scenario.MarkerWritten {
+ t.Fatalf("the marker says %s while the removal waited for the %s lock", again.State, which)
+ }
+ if err := held.Unlock(); err != nil {
+ t.Fatal(err)
+ }
+ res = rm.Pass(r.input())
+ noRemovalErrors(t, res)
+ if len(res.Removed) != 1 {
+ t.Fatalf("after the %s lock was released: removed %v, kept %v, waiting %v", which, res.Removed, res.Kept, res.Waiting)
+ }
+ r.gone(b, m)
+ })
+ }
+}
+
+// TestRemovalWaitsForTheWholeBuild. A build writes a session's files one
+// after another, and its marker last. Until the marker is there, what was
+// written so far can be landed and sent, and the session must stay. Once the
+// build finishes, the rest is landed and sent, nothing is sent twice, and the
+// session goes.
+func TestRemovalWaitsForTheWholeBuild(t *testing.T) {
+ r := newSentRoot(t)
+ sc := removalScenario(t)
+ // A place early in the session for a build to stop at: after the first
+ // call, before the child and the workflow start.
+ sc.Steps[1].Checkpoint = "partway"
+ early := r.build(sc, scenario.Options{Through: "partway"})
+ // As if the build had not reached its last act yet.
+ if err := os.Remove(early.Marker); err != nil {
+ t.Fatal(err)
+ }
+ r.pipeline()
+ rm := r.remover()
+ res := rm.Pass(r.input())
+ noRemovalErrors(t, res)
+ if len(res.Removed) != 0 || !present(r.zone.SessionDir(early.Session)) {
+ t.Fatalf("a session whose build had not finished was removed: %v", res.Removed)
+ }
+ before := sentPerFile(r.rcv)
+ if len(before) == 0 {
+ t.Fatal("nothing was sent while the build was running, so the test exercised nothing")
+ }
+
+ whole := r.build(sc, scenario.Options{})
+ if whole.Session != early.Session {
+ t.Fatalf("the whole build wrote session %s, the early one %s", whole.Session, early.Session)
+ }
+ m := readMarker(t, whole.Marker)
+ removed := false
+ for range 3 {
+ r.pipeline()
+ res := rm.Pass(r.input())
+ noRemovalErrors(t, res)
+ if removed = slices.Contains(res.Removed, whole.Session); removed {
+ break
+ }
+ }
+ if !removed {
+ t.Fatal("three passes after the build finished did not remove the session")
+ }
+ after := sentPerFile(r.rcv)
+ for file, n := range after {
+ if n != 1 {
+ t.Errorf("%s was sent %d times", file, n)
+ }
+ }
+ if len(after) <= len(before) {
+ t.Fatalf("what the build wrote last was never sent: %d files before it finished, %d after", len(before), len(after))
+ }
+ r.gone(whole, m)
+}
diff --git a/tests/scenarios/removed-after-sent.expect.yaml b/tests/scenarios/removed-after-sent.expect.yaml
new file mode 100644
index 0000000..a2e18bf
--- /dev/null
+++ b/tests/scenarios/removed-after-sent.expect.yaml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+checkpoints:
+ final:
+ rounds: 5
+ # Six calls: four on main, one in the helper, one in lint. The helper
+ # and the workflow are each started by an agent call and end with an
+ # output. Only the helper notifies. The main stream has two runs, the
+ # person's and the notification's.
+ kinds: {stream: 3, talk: 3, run: 4, llm.call: 6, tool: 1, agent.call: 2, agent.launch_ack: 2, agent.output: 2, runtime.notification: 1, message.external: 1, message.assistant: 4}
+ talks_on: {main: 1, helper: 1, lint: 1}
+ relations: {starts: 2, ends_with: 2, reports: 1}
+ unresolved: {open: 0}
+ view: {state: verified, rounds: 5, changes: 1, changes_joined: 1, captured_by: {asz-plugin: 1}}
+# A small round budget cuts the chain into several rounds, so a round sent
+# twice, or parsed again after a removal, would show. One expectation holds
+# for both formats. At 3072 bytes both cut five rounds, while at 4096 the
+# claude-code landing cuts four and the sd landing five.
+parse: {max_round_bytes: 3072}
+push:
+ kinds: [transcript, agent_meta, journal, workflow_manifest, workflow_script, changes, round]
diff --git a/tests/scenarios/removed-after-sent.yaml b/tests/scenarios/removed-after-sent.yaml
new file mode 100644
index 0000000..0476403
--- /dev/null
+++ b/tests/scenarios/removed-after-sent.yaml
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# A scenario session, removed once all of it is sent.
+#
+# One session holds every kind of file a removal deletes: the main
+# transcript, a child's transcript and meta file, a workflow's journal,
+# manifest and script, a script filed under another project, and the
+# plugin's output. Every scenario runs the removed_after_sent property. This
+# one gives it every kind at once, and the chain tests stop its removal at
+# every point it reaches.
+title: removed once sent
+steps:
+ - input: fix the timeout and check everything
+ - call:
+ text: Fixing it.
+ tool:
+ name: Bash
+ input: {command: "sed -i 's/= 10/= 30/' server.go"}
+ result: {text: "", after: 500ms}
+ changes:
+ - {path: server.go, before: "package main\n\nvar timeout = 10\n", after: "package main\n\nvar timeout = 30\n"}
+ usage: {in: 2, out: 50, cache_read: 900, cache_write: 100}
+ - call:
+ agent: {name: helper, prompt: look at the tests, steps: [{call: {text: tests look fine, usage: {out: 20}}}], notify: true}
+ - call:
+ workflow:
+ name: verify
+ script_project: -Users-dev-script-filed-elsewhere
+ children:
+ - {name: lint, prompt: run the linter, steps: [{call: {text: lint done}}]}
+ - call: {text: Everything checks out.}