[PR] Remove an idle conversation's landed files, and keep its structure in skeletons [skywalking-ai-sessionizer]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng opened a new pull request, #65:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/65

   Step 4 of retention. A storage root no longer only grows.
   
   ### What it does
   
   `storage.retention`, `-1` and off by default, removes the landed files of a conversation that was sent and has had nothing new land for that long, such as `72h` or `30d`. `asz collect` and `asz server` do it at the end of a pass, after the send.
   
   A conversation's files go together, and only when all of these hold:
   
   - It has an owner in `push.state`: it was sent, or a send was tried.
   - Nothing landed for it within the retention.
   - Its head round covers its newest file, and its parse did not fail this pass.
   - With metrics derived, every file of it is derived.
   - It holds no provider bodies. Exact provider cutting after a removal comes in step 5.
   
   Sending is not waited for. Retention never runs over a scenario root.
   
   ### Skeletons
   
   Each removed file is replaced by `<name>.sk`: Session Data, compressed with gzip. It keeps every record and part in its place, with every identifier the index reads, and no content.
   
   - A changes record keeps its join, its time, and each file's path, operation, sides and line counts, with no diff.
   - An execution record keeps its join, its time and its outcome.
   - A provider body keeps its manifest without segments, and every piece's digest.
   
   The header's new `removed` object records:
   
   - the file's digest, size and record count;
   - when, and by which removal, it was removed;
   - what the receiver confirmed of it, and the owner.
   
   The index rebuilt from skeletons is the index rebuilt from the landed files, byte for byte, so a conversation that becomes active again continues the same chain. A skeleton is never sent.
   
   Measured on 2026-10-13 on one root: 7,235 landed files of 847.5 MB had skeletons of 29.6 MB, 3.5%.
   
   ### Nothing lost
   
   1. `removing.json` records the removal: its generation, the newest sequence it removes, and its phase.
   2. Each skeleton is written and synced, and the error of its first sync is checked. It takes its file's modification time and is synced again. A failed sync stops the removal, and the next pass writes every skeleton again.
   3. Each landed file is deleted only once its skeleton reads whole and records the file's digest. Then each metrics request the receiver has goes; a request not yet sent goes on the first pass after it is. A receipt stays, so a deriver whose state a power loss took back derives the file as before. Every directory of the removed range is synced.
   4. `removing.json` is deleted.
   
   Every pass finishes an unfinished removal before anything lands. No collector lands into a session while its removal is unfinished: it finds the session busy.
   
   ### Readers
   
   | Reader | What it does with a skeleton |
   | --- | --- |
   | index, parse | reads it as the landed file; binds a round to the digest it recorded |
   | metrics | keys and digests by the landed path; tokens from a skeleton, no MCP call |
   | export | never sends it |
   | LangSmith | continues its next position from it |
   | `asz verify` | checks lines and bytes through it; reports removed files apart, never as verified |
   | page | marks the file and its steps `removed`; follows a removal mid-read; keeps what it read for the chain head and the removal generation in `removal.state`, so a separate `asz view` sees a removal |
   | `asz show` | says the record was removed |
   | `asz repack` | refuses a session with skeletons |
   
   Also in this PR: the first commit takes out the owner adoption for `push.state` files an older build wrote. Upgrades from older builds are not supported.
   
   Deferred to later steps:
   
   - **Step 5:** provider body sessions.
   - **Step 6:** the page's banner.
   - **Step 7:** local mode, for never-sent sessions.
   - **Step 8:** `sent_state_retention` and the retired marker.
   
   ### Tests
   
   Each was proven by reverting its fix.
   
   - `tests/chain/retention_test.go`
     - All 34 scenarios run twice, once removing everything after each send. They must give the same sent files, rounds and metrics, chains, indexes (also rebuilt from skeletons only), page file lists, changes and executions, and verify cleanly.
     - Each reason a session is kept.
     - Receiver evidence and generations.
     - Metrics cleanup: a request sent later goes, and receipts stay.
     - A crash at every point of a removal.
     - A failed first sync, and a failed sync after the deletions.
     - Nothing lands during a removal.
     - Empty and wrong skeletons.
     - Metrics derived from skeletons.
     - An untimed file stamped after a removal.
     - The page following a removal, from another process too, and keeping nothing while one is in progress.
   - Unit tests:
     - `pkg/skeleton`: the format.
     - `internal/storage`: listing, digests, the counter, the intent, the removal generation, the fallback open, the busy lock.
     - `internal/verify`: removed files reported, not verified.
     - `internal/view`: whole-record steps, removed bodies not served.
     - `internal/config`: parsing and overflow.
     - `internal/adapters/langsmith`: the next position.
     - `cmd/asz`: the pass line, scenario roots left alone, repack refusing, and `show` and `verify`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Remove an idle conversation's landed files, and keep its structure in skeletons [skywalking-ai-sessionizer]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng merged PR #65:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/65


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org