[PR] Cut a provider body after a removal exactly as with none, and remove sessions that hold them [skywalking-ai-sessionizer]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng opened a new pull request, #66:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/66

   Step 5 of retention. Until now a session holding provider bodies kept its files, because a later body is cut against earlier ones. Such a session is now removed like any other, and a body landed after a removal lands as the same record it would with nothing removed.
   
   ### How
   - **Skeletons hold the cutter's state.** `providerbody.Session.Add` takes a skeleton record in its record's place: identity, digest, size, depth, pieces by digest, and place among the chain tips. It checks the record as it checked the landed one. A skeleton keeps a part's `too_large` state, and the manifest as a reader decodes it, with literal strings taken out and every `part`, `piece` and `copy` kept.
   - **The store** under `<session>/provider/`:
     - `tips/<generation>/` holds every chain tip at a removal, named by the SHA-256 of its record id.
     - `pieces/` holds each removed piece a later body cites, named by its digest.
   
     The removal keeps the tips before it writes its intent. The collector keeps a removed piece before the first body citing it lands. Every file is written to replace what an earlier attempt left, and its directories up to the session's are synced, with errors checked. Each file is checked against its digest when read.
   - **Removal order:**
     1. tips
     2. intent
     3. skeletons
     4. deletes and syncs
     5. reclaim: earlier tips, and the pieces unless a body landed after the range
     6. delete the intent and sync the session directory
   - **`asz verify`** rebuilds every body that is not removed from the store. It counts the bodies a removal in progress deletes. When a removal changed the session during the check, it checks again, and after four tries it says the result is inconclusive.
   
   ### Tests
   - `pkg/skeleton`:
     - Removals at many points and in sequence, with no budget and with a 12 KiB budget. Every body comes out byte-identical. The test requires a removed tip being copied, a removed piece being cited, a removed too-large piece being cited, and a removed tip too large to keep.
     - Odd manifests are judged the same from their skeletons: no segments, literals beside references, `Lit` in capitals, duplicate keys.
     - An identical body under another id does not take a kept tip.
   - `tests/chain`: every scenario with and without removals, at its own budget. A new scenario, `provider-bodies-too-large`. A crash at every point, with verify run before recovery. A resurrected intent, the final sync, a tip that cannot be kept leaving no intent, and an `.sd` build continued after a removal.
   - `internal/storage` and `internal/verify`: store durability, replacement and lookup; a removal during a verify check; a torn read; an inconclusive result.
   
   Each guard was proven by reverting it.
   
   ### Also fixed
   - The chain tests collected with no budget. They now use each scenario's `max_delta_bytes`.
   - `provider-bodies-long` needs 16 KiB for its first request to land whole and be copied.
   - The scenario runner's fold comparison built its plain root at the default budget.
   
   ### Not in this step
   The page cannot yet draw a body landed after a removal that is built from a removed tip or piece. That is step 6.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Cut a provider body after a removal exactly as with none, and remove sessions that hold them [skywalking-ai-sessionizer]

["wu-sheng (via GitHub)" <gi...@apache.org>]

wu-sheng merged PR #66:
URL: https://github.com/apache/skywalking-ai-sessionizer/pull/66


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org