You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2026/10/07 07:31:51 UTC
[PR] feat(index): cut the series index over to the native index and remove the legacy engine (NIDX-03) [skywalking-banyandb]
hanahmily opened a new pull request, #1397:
URL: https://github.com/apache/skywalking-banyandb/pull/1397
Cuts the per-segment series index (`seg-*/sidx`) over to the native index engine and removes the legacy engine from every series, element-index, and Property path. Implements NIDX-03, apache/skywalking#14003.
Design: `docs/design/0.12.0/native-inverted-index/nidx-03-series-index.md`.
Verification report: `docs/design/0.12.0/native-inverted-index/verification/nidx-03-series-cutover/README.md`.
## Contract
- There is no engine flag and no switch between engines.
- The file format is unchanged: ICE v3 segments, v3 snapshot manifests, and segment metadata.
- Rollback is a file property. v0.11.1 opens the directories this build writes and returns the same results.
- Upgrades are whole-cluster. The replication wire and sender are unchanged.
## Commits
1. **Engine primitives (`pkg/index/native`)**
- Prefix and wildcard term-set requests.
- `BatchInsertIfAbsent` with a positive-only presence cache. A batch whose documents already exist publishes nothing.
- `ExternalDedup` with `KeepExisting`.
- `_id` doc values.
- The criteria filter moves out of Property into a shared package, `pkg/index/native/criteria`.
2. **MATCH tokenizing in the query layer.** `pkg/query/logical` tokenizes MATCH values with `nativeanalysis` instead of bluge's analyzers. A test checks that the token sequence matches bluge's.
3. **Series index cutover**
- `seriesIndex` runs on `native.Owner`: insert-if-absent, upsert, identity matching, the criteria filter, index-rule sort, projection, and query-budget charging.
- The Measure planner passes criteria and a field resolver instead of a bluge query.
- Measure and Trace use `RootLeaseFactory`.
- The receiver keeps the existing document on a duplicate `_id`.
- Offline tools move to native: the union-sidx builder, the index-mode copy, and the sidx dump.
- Fixes for pre-existing problems:
- a close error no longer panics; the segment is marked failed;
- closed-segment snapshots hard-link only the newest snapshot and its files, opened strictly;
- index-mode `copyTo` timestamps and versions now stay aligned with their rows.
4. **Legacy removal**
- `pkg/index/inverted` keeps only what the Stream element migration tool needs (NIDX-04).
- Property drops `NativeWriter`, `SwitchIndexWriter`, and the legacy branches.
- The element index writes `_id` doc values.
- Segments build their identifier filter when they are created, so the first lookup after a publish doesn't pay for it.
- A dependency guard test keeps storage, Measure, Trace, and the planner free of `pkg/index/inverted` and bluge.
## Verification
- **Unit tests:** `go build`, `go vet`, and `go test ./banyand/... ./pkg/... ./bydbctl/...` pass (125 packages). `-race` passes on storage, Property, and the native engine. golangci-lint passes with both configs.
- **Integration (local):**
- standalone query: 939/939
- local backup: 3/3
- distributed lifecycle: 8/8
- **Rollback:** `NIDX03_ROLLBACK=1 go test ./test/rollback/nidx03/...` builds v0.11.1 from source. The new build writes the data and v0.11.1 reads it:
- gRPC Measure data, including an update;
- the series index, normal and index mode, plus a SIGKILL crash cut;
- Property data.
An always-run test covers the upgrade direction: Property data in v0.11.1's legacy layout, opened by native.
- **Performance:** 1M series, `-count=6`, compared against v0.11.1. All design targets are met.
| Benchmark | Change |
|---|---|
| Insert of an existing series | −36.5% |
| Exact lookup | −48.5% |
| Prefix lookup | −68.7% |
| Wildcard lookup | −3.8% |
| Index-order search | −81.9% |
| External receive | −10.9% |
External receive allocates about 2× as much memory, because the identifier filter is now built when a segment arrives.
## Not run locally
- the other distributed integration suites and the e2e-v2 docker suites
- Azure and S3 backup, which need credentials
- the dm-delay slow-disk check
`TestRepairServiceDataSyncTwoNodes` failed once in a full local run. It didn't reproduce: it passed 3/3 alone, 8/8 under CPU load at both this tree and the Phase 2 commit, and the next full run passed.
## Out of scope (NIDX-04)
- Moving the Stream element migration tool off `pkg/index/inverted`.
- Removing that package, `pkg/index/analyzer`, and the bluge dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(index): cut the series index over to the native index and remove the legacy engine (NIDX-03) [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #1397:
URL: https://github.com/apache/skywalking-banyandb/pull/1397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(index): cut the series index over to the native index and remove the legacy engine (NIDX-03) [skywalking-banyandb]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1397:
URL: https://github.com/apache/skywalking-banyandb/pull/1397#issuecomment-6033298113
## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1397?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 60.67%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5887324`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/58873240a988166c4e4ffb5bb08b122422f4c092?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 464 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #1397 +/- ##
===========================================
+ Coverage 45.97% 60.67% +14.69%
===========================================
Files 328 35 -293
Lines 55505 6385 -49120
===========================================
- Hits 25520 3874 -21646
+ Misses 27909 2287 -25622
+ Partials 2076 224 -1852
```
| [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1397/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [fodc](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1397/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.67% <ø> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1397?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org