[I] [NIDX-01C] Walk stored documents for migration source reads [skywalking]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily opened a new issue, #14010:
URL: https://github.com/apache/skywalking/issues/14010

   # [NIDX-01C] Walk stored documents for migration source reads
   
   Parent workstream: #14002
   Blocked by: #14009
   
   ## Summary
   
   Add one bounded native match-all/stored-field seam and use it in the source-read sides of Measure index-mode copy and series-index union. Destination publication remains on the retained legacy writer. This live slice adds stored-chunk decoding without adding dictionaries, term queries, doc values, or sorting.
   
   ## Blocked by
   
   NIDX-01B (#14009) must be merged before this work starts.
   
   ## Pre-agreed test seam
   
   Expose one BanyanDB-owned read-only document visitor from `pkg/index/inverted`. It must:
   
   - stream one live document at a time from a pinned committed generation;
   - expose repeated raw `(field name, value bytes)` visits;
   - document that borrowed values expire when the document callback returns; and
   - expose no ICE, Bluge, mmap, FST, or roaring type.
   
   The contract author chooses the smallest concrete Go signature satisfying that shape. Boundary tests target only that visitor. End-to-end tests observe existing migration copy/union outputs, not private stored-chunk functions.
   
   ## Independent fixture
   
   Extend the legacy corpus with live series documents:
   
   - series `101`: `_timestamp=100`, `_version=2`, repeated `color=[blue, green]`, and binary identity `0x010203`;
   - series `202`: `_timestamp=200`, `_version=1`, `color=[red]`; and
   - a deleted series `303`, which must not be visited.
   
   A second source repeats series `101` for the union case. The declared walk is exactly `101, 202`; repeated values stay in the listed order; union output contains `101` once and `202` once.
   
   ## Requirements
   
   R1. The visitor streams only live documents and preserves every repeated field name/value byte sequence exactly. It retains at most one document plus configured decode buffers.
   
   R2. Measure index-mode copy rebuilds series `101` and `202` with the declared timestamps, versions, fields, and identities through the native source visitor, while its destination remains legacy-written.
   
   R3. Series union reads both sources natively and publishes one `101` and one `202`; the existing dedup winner rule is unchanged.
   
   R4. Cancellation stops between documents/chunks, releases files, and publishes no partial destination. Truncated stored chunks, invalid lengths, and decode-limit overflow return the native typed corruption error. CRC32 remains ignored.
   
   ## RED and end-to-end contract
   
   - Boundary RED: visit the checked-in fixture through the new BanyanDB-owned visitor and assert the literal documents and repeated values above.
   - E2E RED: run index-mode copy and series union from legacy-created sources, verify their legacy-written destinations contain exactly `101` and `202`, then run a corrupt-source case and assert no destination is published.
   
   ## Acceptance criteria
   
   - `go test ./pkg/index/inverted/... ./banyand/measure ./banyand/internal/migration -run 'Test(NativeStoredDocumentWalk|E2E.*NativeSource)' -count=1` passes.
   - A call-graph/dependency assertion proves both named source readers use the native visitor and no longer directly open Bluge readers.
   - Cancellation, corrupt-chunk, repeated-value, deletion, and bounded-memory cases pass under `-race`.
   - `make test-ci PKG='./pkg/index/inverted/... ./banyand/measure ./banyand/internal/migration'`, `make build`, and `make pre-push` pass.
   
   ## Scope
   
   Packages: `pkg/index/inverted`, `banyand/measure` migration index-mode source reading, and `banyand/internal/migration` union source reading.
   
   Out of scope: schema kind filtering, term dictionary/posting queries, Property repair, native destination writers, and all online writers.
   
   ## Compatibility and rollback
   
   The native path reads legacy ICE v3/snapshot v3 and emits no index bytes. Roll back only the two source-reader selections; existing destination directories require no conversion.
   
   ## Design
   
   [BDB-NIDX-SPEC-001 revision 0.2 — NIDX-01](https://github.com/apache/skywalking-banyandb/blob/main/docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-01)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [NIDX-01C] Walk stored documents for migration source reads [skywalking]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily closed issue #14010: [NIDX-01C] Walk stored documents for migration source reads
URL: https://github.com/apache/skywalking/issues/14010


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org