You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2026/08/25 06:23:41 UTC

[I] [NIDX-01] Replace the Property shard index with the native implementation [skywalking]

hanahmily opened a new issue, #14002:
URL: https://github.com/apache/skywalking/issues/14002

   # [NIDX-01] Replace the Property shard index with the native implementation
   
   Parent: #13990
   
   ## Summary
   
   Make the native index the production implementation for every existing and new Property shard. This is the first complete vertical slice: it delivers compatible persistence, Property writes and reads, lifecycle handling, and rollback in one merge.
   
   ## Boundary
   
   `banyand/property/db.newShard` selects the native constructor by default for both legacy-created and new shard directories. Property repair uses that same reader. The native surface covers only the analyzers, term codecs, query nodes, and ICE/snapshot sections exercised by Property.
   
   The retained legacy Property constructor is an explicit rollback choice; it is not a second production write path.
   
   ## Requirements
   
   R1. Opening an existing supported Property shard with the native constructor preserves its logical documents, stored fields, delete state, and sort order; no opening-time rewrite or disk-version bump occurs.
   
   R2. An upsert replacing document `D` makes only the replacement visible after publication. A delete of `D` makes `D` absent from exact, boolean, range, and ordered-scan results.
   
   R3. Exact, boolean composition, numeric/date range filtering, stored-field retrieval, and explicit ascending/descending sort return the same logical result as the pinned legacy oracle for the shared fixture corpus.
   
   R4. Property snapshots, callbacks, restart recovery, merge-time expiry, reference-safe garbage collection, repair scan, and file snapshot operate through the native implementation.
   
   R5. A crash before publication leaves the preceding committed generation readable; a crash after publication leaves the complete new generation readable. No committed snapshot references a partial native segment.
   
   R6. Native output retains every required ICE v3 compatibility field. CRC32 fields remain present for compatibility, but the native implementation neither calculates nor validates them.
   
   R7. The pinned legacy Property binary opens, queries, mutates, merges, and restarts a native-produced shard without conversion.
   
   ## Acceptance criteria
   
   - A Property integration test creates a shard with the legacy oracle, then performs native update, query, ordered scan, delete/expiry, callback, repair, merge, restart, and backup; each result matches the fixture's independently enumerated expected documents and stored values.
   - A reciprocal compatibility test creates the shard with the native writer, then runs legacy query, mutation, merge, and restart successfully on the same directory.
   - Fault-injection tests cover at least: segment data written before metadata, metadata written before manifest/snapshot publication, and publication before obsolete-generation GC. Each restart selects a complete committed generation.
   - Corruption tests for the sections Property reads return typed errors and do not panic.
   - `go test ./banyand/property/db ./pkg/index/...` passes, followed by `make build && make pre-push`.
   
   ## Scope
   
   Packages: `banyand/property/db`, `pkg/index`, and the new internal native-index implementation used by the Property constructor.
   
   Out of scope: series identity wildcard/dictionary behavior, Stream element postings or MATCH analyzers, external segment receive, general schema/union/copy/verify/dump APIs, and removal of the legacy adapter.
   
   ## Compatibility and rollback
   
   The on-disk ICE v3/snapshot format is a compatibility contract. Do not add an opening-time migration or a new disk version. Roll back by draining the Property writer and selecting the retained legacy constructor on the same files; startup must fail rather than rewrite if that binary cannot accept the directory.
   
   ## Design
   
   `docs/design/archive/0.12.0/native-inverted-index/delivery-review.html#nidx-01`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [NIDX-01 workstream] Replace bounded read-only index access [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily closed issue #14002: [NIDX-01 workstream] Replace bounded read-only index access
URL: https://github.com/apache/skywalking/issues/14002


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [NIDX-01 workstream] Replace bounded read-only index access [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #14002:
URL: https://github.com/apache/skywalking/issues/14002#issuecomment-5630276996

   All five ordered leaves are merged and closed, which is this workstream's stated closing condition.
   
   | leaf | PR | commit |
   |---|---|---|
   | #14008 — NIDX-01A baseline read-only document count | apache/skywalking-banyandb#1309 | `ba66af94` |
   | #14009 — NIDX-01B committed generations with deletions | apache/skywalking-banyandb#1316 | `eeb3b927` |
   | #14010 — NIDX-01C stored document walks for migration sources | apache/skywalking-banyandb#1318 | `411a3941` |
   | #14011 — NIDX-01D exact-term schema walks | apache/skywalking-banyandb#1325 | `59805944` |
   | #14012 — NIDX-01E Property repair search-after | apache/skywalking-banyandb#1335 | `31b32ca2` |
   
   The bounded read-only surface is on `main` at `31b32ca2`: committed-generation open, visible count, match-all stored walk, exact term/OR, deletion masks, required scalar decoding, and ascending Property repair search-after. The native reader package imports no retired index dependency and reads a shard directory directly.
   
   Closing this unblocks #14007 (NIDX-02) for just-in-time decomposition.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org