You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2026/09/15 05:56:02 UTC
(skywalking-ai-sessionizer) branch main updated: Collect every 10 minutes by default, and keep 5 seconds for tests (#9)
This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/skywalking-ai-sessionizer.git
The following commit(s) were added to refs/heads/main by this push:
new e6ef434 Collect every 10 minutes by default, and keep 5 seconds for tests (#9)
e6ef434 is described below
commit e6ef4344c8008c2ca4417bd3a1773dfcbcc455dc
Author: 吴晟 Wu Sheng <wu...@foxmail.com>
AuthorDate: Tue Sep 15 13:55:56 2026 +0800
Collect every 10 minutes by default, and keep 5 seconds for tests (#9)
Every pass that finds new records lands new files, and a landed file is
one log record when it is sent. At the 5 second default, a watching
collector wrote a small file for every stream that moved, every 5
seconds. Nothing in the docs said to raise it.
- The collector's default interval is 10 minutes, in asz.yaml and when
no configuration is given. asz collect and asz server still make a
pass when they start.
- A scenario build writes interval: 5s into the configuration it
creates, so a collector beside a feed picks up each session as it
arrives. A directory an earlier build wrote without an interval is
brought up to date instead of refused.
- The configuration page gains a section on choosing an interval, and
the quick start, command line, container image and scenario pages,
the collect skill and the 0.4.0 changelog say 10 minutes.
---
.claude/skills/collect/SKILL.md | 4 +--
asz.yaml | 9 +++++--
docs/en/changes/changes.md | 11 ++++++++
docs/en/guides/scenario.md | 3 ++-
docs/en/setup/command-line.md | 2 +-
docs/en/setup/configuration.md | 22 +++++++++++----
docs/en/setup/container-image.md | 3 ++-
docs/en/setup/quick-start.md | 6 +++--
internal/config/config.go | 12 ++++++---
internal/scenario/build.go | 55 +++++++++++++++++++++++---------------
internal/scenario/scenario_test.go | 26 ++++++++++++++++++
11 files changed, 114 insertions(+), 39 deletions(-)
diff --git a/.claude/skills/collect/SKILL.md b/.claude/skills/collect/SKILL.md
index 04fb264..cf36b49 100644
--- a/.claude/skills/collect/SKILL.md
+++ b/.claude/skills/collect/SKILL.md
@@ -53,7 +53,7 @@ Default storage root is `./data`, relative to the working directory.
```sh
./bin/asz collect -once # single pass; the backfill path
-./bin/asz collect # watch mode, 5s interval
+./bin/asz collect # watch mode: a pass now, then every 10 minutes
```
To land somewhere else, write a config and pass it:
@@ -74,7 +74,7 @@ adapters:
```
Without `-config`, a command run from the repository root reads the `asz.yaml` there. Its
-values are the built-in defaults: storage root `./data`, watch mode, a 5 second interval. Pass a
+values are the built-in defaults: storage root `./data`, watch mode, a 10 minute interval. Pass a
file like the one above when you want a single backfill pass into another directory.
## 5. Read the result line
diff --git a/asz.yaml b/asz.yaml
index 10a3b31..d7a3090 100644
--- a/asz.yaml
+++ b/asz.yaml
@@ -69,8 +69,13 @@ adapters:
# over history that already exists. asz server runs the same pipeline
# beside the page, and shows the last and the next refresh on its
# list page.
+ #
+ # Both commands make a pass when they start, then one every interval.
+ # A pass lands what moved as new files, so a short interval lands many
+ # small files, and each file is one log record when it is sent. Keep
+ # a few seconds for tests; scenario feeds write their own.
mode: watch
- interval: 5s
+ interval: 10m
# The largest landed file the collector writes, in bytes. A large
# catch-up is split into several files rather than one enormous one,
@@ -115,7 +120,7 @@ adapters:
collector:
mode: watch
- interval: 5s
+ interval: 10m
max_delta_bytes: 2097152
parse:
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 9d41cee..9fc45cf 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -1,3 +1,14 @@
# Changes in 0.4.0
> In development, not yet released. `tools/release.sh prepare 0.4.0` removes this note.
+
+## Collection
+
+- The collector's default `interval` is 10 minutes, in `asz.yaml` and when no configuration is
+ given; it was 5 seconds. Every pass that finds new records lands new files, so a period of
+ seconds wrote many small files, and each one is a log record when sent. `asz collect` and
+ `asz server` still make a pass when they start. A configuration that sets `interval` keeps its
+ value. [Configuration](../setup/configuration.md#choosing-an-interval) explains the choice.
+- A scenario build writes `interval: 5s` into the configuration it creates, so a collector beside a
+ feed still picks up each session as it arrives. A directory an earlier build wrote without an
+ interval is brought up to date instead of refused.
diff --git a/docs/en/guides/scenario.md b/docs/en/guides/scenario.md
index 76f70b9..1439bef 100644
--- a/docs/en/guides/scenario.md
+++ b/docs/en/guides/scenario.md
@@ -76,7 +76,8 @@ asz scenario build tests/scenarios/assembly.yaml --format claude-code --out DIR
asz server -config DIR/asz.yaml # collect, parse and serve, on the interval
```
-`server` refreshes on the interval because the configuration the build writes says `mode: watch`.
+`server` refreshes every 5 seconds because the configuration the build writes says `mode: watch`
+and `interval: 5s`. The product default is 10 minutes, which would make a feed look stopped.
To feed a receiver instead of a page, name it under `export.otlp` and run `asz collect`, which
sends at the end of every period. With a receiver named, under either command, each session of the
feed is removed once all of it is sent, or later with `--remove`.
diff --git a/docs/en/setup/command-line.md b/docs/en/setup/command-line.md
index 2fa0d1f..e892d8d 100644
--- a/docs/en/setup/command-line.md
+++ b/docs/en/setup/command-line.md
@@ -312,7 +312,7 @@ a receiver. Nothing needs a restart. Every request for a conversation lists its
and folds again when the head round has moved. A landed file that arrives after the fold is found
by a new scan of the session directory. That scan runs at most once a second, so a request for a
sequence that does not exist does not walk the directory every time. The root itself is checked on
-the collector interval, 5 seconds by default. `/api/status` reports as `last_refresh` the last time
+the collector interval, 10 minutes by default. `/api/status` reports as `last_refresh` the last time
a head round or the size of its file changed, and the list page reloads when that time moves.
A round file is created under its final name and written in place, so a listing can show a round
diff --git a/docs/en/setup/configuration.md b/docs/en/setup/configuration.md
index 0b3c3c8..af1d048 100644
--- a/docs/en/setup/configuration.md
+++ b/docs/en/setup/configuration.md
@@ -18,7 +18,7 @@ adapters:
- /private/tmp/**
collector:
mode: watch
- interval: 5s
+ interval: 10m
max_delta_bytes: 2097152
```
@@ -75,9 +75,21 @@ are the tool's, not yours, which is why they are excluded by default.
| Key | Default | Meaning |
| --- | --- | --- |
| `mode` | `watch` | `watch` polls the source continuously. `once` makes a single pass, which is the backfill path over history that already exists. `asz collect` then exits, and `asz server` goes on serving what that pass produced. `-once` on the command line overrides the file. |
-| `interval` | `5s` | How long the pipeline sleeps between passes in watch mode. It is the whole period: `asz collect` and `asz server` land, parse and send once per interval. |
+| `interval` | `10m` | How long the pipeline sleeps between passes in watch mode. It is the whole period: `asz collect` and `asz server` land, parse and send once per interval. Both make a pass when they start, then one every interval. |
| `max_delta_bytes` | `2097152` | How much of a growing source the collector lands in one `.sd` file, 2 MiB. A large catch-up is split into several files. When the next source line is longer than the budget, the collector reads to the end of the source instead. That file then holds the long line and every complete line after it. A file travels whole as one log record, so the largest record a receiver has to accept is the largest file, not the budget. A change applies to new files only. ` [...]
+### Choosing an interval
+
+Keep the interval in minutes. A pass lands what moved since the last pass, and a landed file is
+never appended to, so every pass that finds new records writes new files. A short interval
+therefore writes many small files. Each landed file is one log record when it is
+[sent](export-otlp.md), so it also means many small records at the receiver. A longer interval
+writes fewer, larger files, up to `max_delta_bytes` each, and the page and the receiver see new
+data that much later.
+
+A few seconds is a value for tests. A [scenario build](../guides/scenario.md) writes `5s` into the
+configuration it creates, so a collector beside a feed picks each session up as it arrives.
+
### Several local adapters
`collector` is set on each adapter, but `asz collect` and `asz server` run one pipeline for all
@@ -93,9 +105,9 @@ settings combine:
A disabled adapter does not count. The receiver, `claude-code-otlp`, takes no `collector` block,
so it does not count either. When it is the only adapter enabled, `asz collect` sends what it
-lands to `export.otlp.endpoint` every 5 seconds, the compiled default.
+lands to `export.otlp.endpoint` every 10 minutes, the compiled default.
-Both commands print the result when they start, on their `source` line: `(every 5s)` when the
+Both commands print the result when they start, on their `source` line: `(every 10m0s)` when the
pipeline watches, `(once)` when it does not. `asz view` collects nothing, but it checks the root
for new rounds at the same shortest interval.
@@ -155,7 +167,7 @@ adapters:
- /private/tmp/**
collector:
mode: watch
- interval: 5s
+ interval: 10m
max_delta_bytes: 2097152
```
diff --git a/docs/en/setup/container-image.md b/docs/en/setup/container-image.md
index 2242ba4..afdc871 100644
--- a/docs/en/setup/container-image.md
+++ b/docs/en/setup/container-image.md
@@ -92,7 +92,8 @@ docker run --rm -p 8787:8787 \
ghcr.io/apache/skywalking-ai-sessionizer:latest
```
-The default command is `server`, so this lands, parses and serves on the collector's interval. To
+The default command is `server`, so this lands, parses and serves on the collector's interval: once
+when it starts, then every 10 minutes unless the configuration sets another. To
collect without serving a page, put `collect` after the image name.
The container user is not the host user, so on Linux the storage root must be writable by it.
diff --git a/docs/en/setup/quick-start.md b/docs/en/setup/quick-start.md
index 1a6c4e8..e589015 100644
--- a/docs/en/setup/quick-start.md
+++ b/docs/en/setup/quick-start.md
@@ -106,8 +106,10 @@ Open `http://127.0.0.1:8787`. The list page shows every conversation; a conversa
its talks, its execution streams, the flow on a time axis, and the evidence behind every step.
On a machine with Claude Code, `asz server` alone is enough. It runs the collector and the parser
-itself, every 5 seconds by default, and the list page shows when the data was last refreshed and
-when it will be next. The two steps above are worth running once to see what each does.
+itself, once when it starts and then every 10 minutes by default, and the list page shows when the
+data was last refreshed and when it will be next. A shorter
+[interval](configuration.md#choosing-an-interval) shows new data sooner and writes more, smaller
+files. The two steps above are worth running once to see what each does.
## Check
diff --git a/internal/config/config.go b/internal/config/config.go
index 63232dd..e8c3768 100644
--- a/internal/config/config.go
+++ b/internal/config/config.go
@@ -199,6 +199,12 @@ func (a *Adapter) UnmarshalYAML(node *yaml.Node) error {
return nil
}
+// DefaultInterval is the period between passes in watch mode. Every pass
+// lands what moved as new files, so a short period lands many small files,
+// and each file is one log record when it is sent. Tests and scenario feeds
+// set their own shorter period.
+const DefaultInterval = 10 * time.Minute
+
// Collector controls collection cadence for one adapter.
type Collector struct {
// Mode is "watch" (poll continuously) or "once" (single pass, then exit).
@@ -244,7 +250,7 @@ func Default() *Config {
MetricsLookback: "24h",
Collector: Collector{
Mode: ModeWatch,
- Interval: 5 * time.Second,
+ Interval: DefaultInterval,
MaxDeltaBytes: 2 << 20,
},
}, {
@@ -264,7 +270,7 @@ func Default() *Config {
Exclude: []string{"/private/tmp/**"},
Collector: Collector{
Mode: ModeWatch,
- Interval: 5 * time.Second,
+ Interval: DefaultInterval,
MaxDeltaBytes: 2 << 20,
},
}},
@@ -354,7 +360,7 @@ func (c *Collector) applyDefaults() {
c.Mode = ModeWatch
}
if c.Interval <= 0 {
- c.Interval = 5 * time.Second
+ c.Interval = DefaultInterval
}
if c.MaxDeltaBytes <= 0 {
c.MaxDeltaBytes = 2 << 20
diff --git a/internal/scenario/build.go b/internal/scenario/build.go
index 2bd4601..db19142 100644
--- a/internal/scenario/build.go
+++ b/internal/scenario/build.go
@@ -125,36 +125,51 @@ func Build(sc *Scenario, format Format, out string, opts Options) (*Built, error
// A one-shot build leaves the collector in once mode, because the source
// stops growing when the build returns. A feed keeps writing new sessions,
// so its configuration says watch, and asz collect or asz server beside it
-// picks each one up on its own interval.
+// picks each one up within feedInterval. The product default is minutes,
+// which would leave a feed looking stopped.
func writeConfig(out, source string, watch bool) (string, error) {
path := filepath.Join(out, "asz.yaml")
mode, other := "once", "watch"
if watch {
mode, other = "watch", "once"
}
- text := configText(out, source, mode)
+ text := configText(out, source, mode, feedInterval)
old, err := os.ReadFile(path)
if err != nil {
return path, os.WriteFile(path, []byte(text), 0o644)
}
- switch {
- // A file that starts with what the build writes is the build's, with
- // whatever a person appended, such as the export block for a push.
- case strings.HasPrefix(string(old), text):
+ if strings.HasPrefix(string(old), text) {
+ // A file that starts with what the build writes is the build's, with
+ // whatever a person appended, such as the export block for a push.
return path, nil
+ }
// The same build in the other collector mode, which is what adding or
- // dropping --every does to a directory that already holds sessions.
- // Rewrite the block and keep what was appended to it.
- case strings.HasPrefix(string(old), configText(out, source, other)):
- kept := old[len(configText(out, source, other)):]
- return path, os.WriteFile(path, append([]byte(text), kept...), 0o644)
- default:
- return "", errors.New("scenario: " + path + " exists and is not this build's; use another --out")
+ // dropping --every does to a directory that already holds sessions, or
+ // a file an earlier build wrote without an interval. Rewrite the block
+ // and keep what was appended to it.
+ for _, prior := range []string{
+ configText(out, source, other, feedInterval),
+ configText(out, source, mode, ""),
+ configText(out, source, other, ""),
+ } {
+ if strings.HasPrefix(string(old), prior) {
+ return path, os.WriteFile(path, append([]byte(text), old[len(prior):]...), 0o644)
+ }
}
+ return "", errors.New("scenario: " + path + " exists and is not this build's; use another --out")
}
-// configText is the configuration for one collector mode.
-func configText(out, source, mode string) string {
+// feedInterval is the collector period a scenario build writes. It is short
+// so a watching collector picks up a feed's sessions as they arrive.
+const feedInterval = "5s"
+
+// configText is the configuration for one collector mode. An empty interval
+// is the form builds wrote before they set one.
+func configText(out, source, mode, interval string) string {
+ collector := " collector:\n mode: " + mode + "\n"
+ if interval != "" {
+ collector += " interval: " + interval + "\n"
+ }
return fmt.Sprintf(`# Written by asz scenario build. The storage root is this directory; the
# adapter's source is the source directory beside it, which an sd build
# leaves empty.
@@ -168,19 +183,15 @@ adapters:
# no look-back: a scenario is history by construction.
metrics: true
metrics_lookback: none
- collector:
- mode: %s
- - name: claude-code-changes
+%s - name: claude-code-changes
enabled: true
source_root: %s/plugins/data
- collector:
- mode: %s
-# To export the session with asz push, name the receiver:
+%s# To export the session with asz push, name the receiver:
# export:
# otlp:
# protocol: grpc
# endpoint: 127.0.0.1:11800
-`, out, source, mode, source, mode)
+`, out, source, collector, source, collector)
}
func mkdirAll(dir string) error { return os.MkdirAll(dir, 0o755) }
diff --git a/internal/scenario/scenario_test.go b/internal/scenario/scenario_test.go
index cfd21a7..d9a5364 100644
--- a/internal/scenario/scenario_test.go
+++ b/internal/scenario/scenario_test.go
@@ -200,6 +200,11 @@ func TestBuildSwitchesCollectorModeAndKeepsWhatWasAppended(t *testing.T) {
if !strings.Contains(string(after), "mode: watch") {
t.Fatal("the collector is not in watch mode after a feed build")
}
+ // The product default is minutes, so a feed must say its own period or
+ // a watching collector beside it looks stopped.
+ if !strings.Contains(string(after), "interval: "+feedInterval) {
+ t.Fatal("a feed build did not write its collector interval")
+ }
if !strings.HasSuffix(string(after), appended) {
t.Fatal("what was appended to the configuration was lost")
}
@@ -207,4 +212,25 @@ func TestBuildSwitchesCollectorModeAndKeepsWhatWasAppended(t *testing.T) {
if _, err := Build(sc, FormatClaudeCode, out, Options{At: at}); err != nil {
t.Fatalf("a one-shot build into a feed's directory was refused: %v", err)
}
+
+ // A directory an earlier build wrote, before builds set an interval, is
+ // still this build's: the block gains the interval and keeps the rest.
+ current, err := os.ReadFile(b.Config)
+ if err != nil {
+ t.Fatal(err)
+ }
+ earlier := strings.ReplaceAll(string(current), " interval: "+feedInterval+"\n", "")
+ if err := os.WriteFile(b.Config, []byte(earlier), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := Build(sc, FormatClaudeCode, out, Options{At: at, Watch: true}); err != nil {
+ t.Fatalf("a feed into an earlier build's directory was refused: %v", err)
+ }
+ rewritten, err := os.ReadFile(b.Config)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !strings.Contains(string(rewritten), "interval: "+feedInterval) || !strings.HasSuffix(string(rewritten), appended) {
+ t.Fatalf("an earlier build's configuration was not brought up to date:\n%s", rewritten)
+ }
}