(skywalking-banyandb) branch agent/curate-0-11-changes created (now 6a44d0042)

[ha...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

hanahmily pushed a change to branch agent/curate-0-11-changes
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git


      at 6a44d0042 docs: curate 0.11.0 release notes

This branch includes the following new commits:

     new 6a44d0042 docs: curate 0.11.0 release notes

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

(skywalking-banyandb) 01/01: docs: curate 0.11.0 release notes

[ha...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

hanahmily pushed a commit to branch agent/curate-0-11-changes
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git

commit 6a44d0042a270d0aac9b236d11888ef2922a647c
Author: Gao Hongtao <ha...@gmail.com>
AuthorDate: Mon Aug 17 02:45:40 2026 +0000

    docs: curate 0.11.0 release notes
---
 CHANGES.md | 592 +++----------------------------------------------------------
 1 file changed, 23 insertions(+), 569 deletions(-)

diff --git a/CHANGES.md b/CHANGES.md
index 8ebf6b940..50c49c954 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -8,17 +8,14 @@ Release Notes.
 - Add bounded trace plugin overhead metrics: chain decision batches by result, per-plugin execution count and duration, batch trace-count distribution, and per-plugin fail-open bypasses by reason. The configured `plugin_name` distinguishes links in a multi-plugin chain; segment, shard, and schema labels remain absent. The catalog works with `pipeline_traces_evaluated` for per-plugin time/calls per trace, latency percentiles, timeout/circuit-open rates, and link-health monitoring. The Pha [...]
 - Add trace **finalization sampling** (`PIPELINE_EVENT_FINALIZE`), a best-effort background backstop for the in-merge trace retention filter. A single node-wide, concurrency-1 scanner periodically sweeps cooled segments (`segEnd < now − finalize_grace`, reopening idle-closed segments via `SelectSegments(reopenClosed=true)`) and force-merges each shard's un-finalized parts through the group's registered sampler chain, reusing the hot merge path (`mergeParts` + per-sidx `Merge` + the shard [...]
 - Bound the trace merge **drop-set** so a large sampled merge cannot exhaust the heap. The merge-wide dropped-trace-ID collector was unbounded: because a shard's first finalize round selects every cooled part into one merge, an 18M-drop set reaches ~1.3 GiB live and ~2.6 GiB reserved heap (append doubling plus a single contiguous lookup index), inside a process also serving queries. The bound is applied to the sampling **decision**, not to the pruning predicate: once a merge's drop set i [...]
-- Add two catalogs to the queue batch-write metrics so traffic is comparable on both ends: a per-batch-stream **batch catalog** (`total_batch_started`/`total_batch_finished`/`total_batch_latency`, buckets to ~300s) on `queue_pub`/`queue_sub` and the `lifecycle_migration` mirror, and a per-message **message catalog** (`total_message_started`/`total_message_finished`) on `queue_sub` (the publisher's existing `total_*` already counts per message). All existing `total_*` series are unchanged [...]
-- Redesign the queue (`queue_pub`/`queue_sub`) metrics around a uniform model: keep only `total_started`, `total_finished`, `total_latency` (now a histogram) and `total_err`, plus file-sync-only `sent_bytes` (pub) / `received_bytes` (sub). Replace the `topic` label with `operation` (`batch-write`/`file-sync`/`query`/`control`) and `group`, add an `error_type` label on `total_err`, and add remote-endpoint labels (`remote_node`/`remote_role`/`remote_tier`) so the liaison↔data (hot/warm/col [...]
+- Redesign queue (`queue_pub`/`queue_sub`) metrics around a uniform model with per-message and per-batch-stream catalogs. Use `operation` and `group` labels, add `error_type` and remote-endpoint labels, and propagate sender identity and group on the wire. [Breaking Change] Previous queue metric families, chunk-ordering metrics, and the `topic` label are removed; update dashboards and alerts.
 - Stamp the lifecycle's tier-migration publisher's identity onto the wire so the receiving data node records a non-empty `remote_node`/`remote_role`/`remote_tier` on its `banyandb_queue_sub_total_finished` series. The lifecycle's `parseGroup` resolves the lifecycle's self identity by matching the lifecycle pod's hostname (POD_NAME via the K8s downward API, falling back to `os.Hostname()` — same precedence as `nativeNodeContext` at `banyand/backup/lifecycle/service.go`) against the data-n [...]
-- Add `banyandb_lifecycle_last_run_timestamp_seconds` and `banyandb_lifecycle_last_run_success` gauges to the lifecycle service for at-a-glance health monitoring. `last_run_timestamp_seconds` records the wall-clock epoch (in seconds) of the most recent migration cycle; `last_run_success` is `1` on a nil error and `0` otherwise. Both are stamped by a `defer` at the end of `action()` so every return path (success, error, recovered panic) updates the pair atomically — dashboards can pin an  [...]
-- Refactor the lifecycle cycle-level metrics (`banyandb_lifecycle_cycles_total`, `banyandb_lifecycle_last_run_timestamp_seconds`, `banyandb_lifecycle_last_run_success`) to carry labels `remote_node`, `remote_role`, `remote_tier`, `group`. The label form mirrors the per-message `banyandb_lifecycle_migration_*` family emitted by the queue/pub lifecycle publisher, but the two families describe DIFFERENT things: the cycle-level series describe the SENDER (the lifecycle pod's co-located data  [...]
+- Add labeled lifecycle health metrics (`banyandb_lifecycle_cycles_total`, `banyandb_lifecycle_last_run_timestamp_seconds`, `banyandb_lifecycle_last_run_success`) keyed by `remote_node`, `remote_role`, `remote_tier`, and `group`; stamp the last-run gauges atomically at cycle end and remove their previous label tuple. [Breaking Change] Update dashboards and alerts for the labeled series.
 - Remove `banyandb_lifecycle_self_identity_resolution_total`. The regression-detection role moves to the now-labeled `banyandb_lifecycle_cycles_total{remote_node!=""}` (an empty `remote_node` series means the registry match failed for every group, the bug the old counter caught), plus the existing receiver-side count of empty `remote_node` on lifecycle `banyandb_queue_sub_total_finished` series. The wire-level `cluster.v1.SendRequest` sender-identity fields are unchanged.
 - Vectorized measure query path is now enabled by default. The columnar pipeline replaces per-row protobuf serialization in `NewMIterator`, cutting allocations and ns/op for scan-heavy measure queries; gRPC wire format (`*measurev1.InternalDataPoint`) is byte-identical. Single-node coverage is complete: scan, GroupBy+Agg via `BatchAggregation`, scalar reduce (`Agg` without `GroupBy`), raw `GroupBy` (without `Agg`), implicit projection coverage for GroupBy/Agg fields, `TopN`/`BottomN`, `o [...]
-- Vectorized **stream** and **trace** query paths are now enabled by default, joining measure. `--stream-vectorized-enabled` and `--trace-vectorized-enabled` now default to `true`. Validated on two axes. A 48h single-node containerized soak per engine covers the compute path (stream 575/575 parity cycles and 84.75M spans; trace parity pass and 84.6M spans; zero memory alerts, no restarts, and TTL reclamation observed releasing memory proportionally to the data it cached). A 6h soak per e [...]
-- Add the vectorized **stream** query path. Timestamp-ordered (asc/desc), index-ordered, criteria-filtered, and multi-group queries dispatch through the columnar pipeline (scan -> batch -> merge -> distinct -> limit) with a native columnar liaison<->data wire frame replacing per-element protobuf. Shapes the vec dispatch does not recognize -- and index-ordered queries whose sort tag is absent from the projection, where the OrderKey column cannot be populated -- fall back to the row path,  [...]
+- Enable vectorized stream and trace queries by default. Stream uses the columnar pipeline and native liaison-to-data frame for supported timestamp-, index-, criteria-, and multi-group queries; unsupported shapes and traced queries fall back to row or protobuf execution. Roll back with `--stream-vectorized-enabled=false` or `--trace-vectorized-enabled=false`.
 - Add `banyandb_vec_frame_encoded{engine}` and `banyandb_vec_frame_decoded{engine}` gauges (engine: `measure`/`stream`/`trace`), reporting how many query responses this process has put on the wire as a native columnar frame and how many it has decoded from one. Which wire format a node is actually speaking was otherwise invisible from outside the process, and the 0.11 upgrade-ordering rule depends on exactly that: a data node emits frames only when it is distributed AND its engine flag i [...]
-- [Breaking Change] Enabling a vectorized path by default changes the required rolling-upgrade order: **upgrade "liaison" nodes BEFORE "data" nodes**, the reverse of the general strategy. Each `--*-vectorized-enabled` flag selects the liaison<->data wire format as well as the query engine -- a flag-on distributed data node emits a native columnar frame. A 0.11 liaison decodes both frames and protobuf (dispatching per message on the frame magic byte), but an older liaison has no frame dec [...]
+- [Breaking Change] Upgrade liaison nodes before data nodes when rolling out vectorized query paths. A 0.11 liaison accepts both columnar frames and protobuf, but an older liaison cannot decode frames emitted by an upgraded data node. Standalone deployments are unaffected.
 - Add validation to ensure Measure's ShardingKey contains all Entity tags to guarantee entity locality.
 - Organize access logs under a dedicated "accesslog" subdirectory to improve log organization and separation from other application data.
 - Collect BanyanDB data on e2e test failure for CI debugging.
@@ -26,15 +23,14 @@ Release Notes.
 - Sync lifecycle e2e test from SkyWalking stages test.
 - Add `noDuplicates` verification to all e2e expected files to detect duplicate data in query results.
 - Add a program-generated trace query integration-test framework under `test/cases/trace/cmd/{generate,capture}`: layered case generation (criteria leaves, AND/OR trees, and feature-pairwise across the traceID-lookup and order-based query modes), env-gated golden capture, and a shared `SeedAll` seeder — mirroring the measure test-case framework.
-- Add periodic health check for property schema connection.
 - Persist segment end time in per-segment metadata so boundaries don't shift across restarts or config changes.
 - Introduce fair fast/slow lane scheduling for trace part merges to prevent short merges from being blocked by long-running merges; expose queue wait time as `total_merge_queue_latency`.
-- [Breaking Change] Remove etcd components. The property-based schema registry is now the only supported mode. 
-  - All `--etcd-*` CLI flags have been removed. 
+- [Breaking Change] Remove etcd components. The property-based schema registry is now the only supported mode.
+  - All `--etcd-*` CLI flags have been removed.
   - The `--namespace` CLI flag has been removed (it previously configured the etcd key prefix).
-  - The `--node-discovery-mode` flag no longer accepts `etcd` (supported values: `none`, `dns`, `file`). 
+  - The `--node-discovery-mode` flag no longer accepts `etcd` (supported values: `none`, `dns`, `file`).
   - The `--schema-registry-mode` flag only accepts `property`.
-- implement panic diagnostics and FODC crash reporting pipeline.
+- Implement panic diagnostics and the FODC crash reporting pipeline.
 - Schema consistency (Phase 1): introduce client-observable revision and propagation primitives. All gates are opt-in and zero-valued requests preserve prior behavior.
   - Add `mod_revision` to Group / IndexRule / IndexRuleBinding / TopNAggregation Create and Update responses.
   - Add `delete_time` to all `*ServiceDeleteResponse` messages so clients can observe tombstones.
@@ -47,39 +43,18 @@ Release Notes.
   - Add tombstone retention/GC (default 7 days, configurable via `--schema-server-tombstone-retention`) with a per-cache count cap to bound memory under bulk deletes.
   - Reject `Create` with `updated_at <= tombstone.delete_time` to prevent replayed creates from overwriting newer deletes.
   - Guard `pkg/schema/cache` against out-of-order `EventDelete` events; expose monotonic `LatestModRevision` watermark.
-- Schema consistency (Phase 2 in progress): cluster-wide barrier. Internal-only; no client-facing surface impact yet.
-  - Add `NodeSchemaStatusService` (`GetMaxRevision`, `GetKeyRevisions`, `GetAbsentKeys`) registered on every cluster member that holds a schema cache, so peer liaisons and data nodes can be probed identically by the barrier fan-out (#1108).
-  - Extend `queue.Client` with `NewNodeSchemaStatusClient(node)` so the barrier fan-out can borrow the existing tier1/tier2 connection pools instead of opening a parallel mesh (#1109).
-  - `AwaitRevisionApplied` now fans out across the receiving liaison's frozen tier1 (peer-liaison) + tier2 (data-node) Active set, probing each member in parallel via `GetMaxRevision` with shared per-call deadline. Cross-version peers returning `codes.Unimplemented` are treated as ready so partial-upgrade clusters do not deadlock; transient RPC errors count as per-iteration laggards. Empty Active set fails fast with `codes.Unavailable`.
-  - Frozen-snapshot mid-call semantics: members that transition `Active → Evictable` during a call are dropped from subsequent probes and surfaced once as a `NodeLaggard{reason="evicted_during_poll"}`; members that disappear from the route table altogether are dropped silently; late joiners are excluded from the watched set until the next call. Adds `reason` field (5) to `NodeLaggard` proto.
-  - `AwaitSchemaApplied` and `AwaitSchemaDeleted` follow the same fan-out shape using `GetKeyRevisions` / `GetAbsentKeys` respectively, with per-node calls chunked at 1000 keys and a shared call-wide deadline (no equal-slice division across chunks). Per-node laggards carry the per-member `missing_keys` / `still_present_keys` they observed.
-  - First-attempt re-enable of the four Phase-1-deferred distributed specs confirmed all four still flake under the cluster barrier alone (second run reproduced the `group not found` race). Preliminary guards added; later superseded by the successful re-enable below.
-  - Add `pkg/schema/registry.NodeRepoRegistry`, the per-node aggregator that routes barrier and node-status RPC lookups to the same per-service `pkg/schema.schemaRepo` instances the data-node executor consults via `LoadGroup` / `LoadResource`. Each banyand service (measure / stream / trace) registers its `schemaRepo` here during PreRun under a kind bitmask covering `KindGroup` + the catalog's primary kind + `KindIndexRule` + `KindIndexRuleBinding`. `metadata.Service` exposes a `NodeRepoR [...]
-  - Repoint `NodeSchemaStatusService` at the `NodeRepoRegistry` for executor-tracked kinds: `GetMaxRevision` reads `schemaCache.notifiedModRevision` directly and `GetKeyRevisions` / `GetAbsentKeys` route per-key lookups through the per-service `schemaRepo` aggregator. TopN / Property keys still consult the property `schemaCache`. This closes the `SendMetadataEvent` eventCh-retry leak where the schemaCache watermark advanced before `schemaRepo.groupMap` applied the event, so the cluster b [...]
-  - Write gate (`validateWriteRequest` for measure / stream / trace) and per-group query gate (`checkQueryGate`) read `cacheRev` through the `NodeRepoRegistry` rather than the liaison `entityRepo` locator. The locator still answers existence checks (`STATUS_NOT_FOUND` signal) and downstream navigation; only the revision scalar moves. Net contract: if `AwaitRevisionApplied(R)` on a node returns `applied=true`, the write gate, query gate, and downstream executor on that same node all see ≥ [...]
-  - Re-enable  /  /  /  in distributed mode (final pass). Distributed schema integration suite reports `Ran 28 of 28 Specs, 0 Skipped`.
-  - Add observability for the schema-consistency cluster (Step 2.7, ): `schema_await_revision_applied_duration_seconds{result}`, `schema_await_schema_applied_duration_seconds{result}`, `schema_await_schema_deleted_duration_seconds{result}` track barrier latency by outcome (`applied` / `timeout` / `invalid_argument` / `error`); `schema_barrier_laggard_nodes_total{barrier,role,node}` decodes the `<role>-<Metadata.Name>` laggard identifier so dashboards can break out which member fell behin [...]
-  - Add the schema-barrier CP-6 SLO load harness (Step 2.8) under `test/load/schema_barrier/`, runnable via `make load-test-barrier`. The harness brings up an in-process 3 data node + 1 liaison cluster, drives 100 concurrent `AwaitRevisionApplied` callers + 10 `Group.Update` ops/sec, and reports p50 / p95 / p99 / max from client-side per-call duration after a 1-minute warm-up + 5-minute measurement window. Client-side latency is bounded above by the server-side histogram so the SLO check [...]
-  - Land `pkg/test/setup.PauseDataNodeWatch` / `ResumeDataNodeWatch` (Step 1.0 follow-up): the helpers replace the `ErrWatchControlNotImplemented` stub with a working hook into `property.SchemaRegistry` so cluster-only specs can drive a single data node to fall behind the cluster while the rest stays in sync. The data node's `handleWatchEvent`, `processInitialResourceFromProperty`, and `handleDeletion` paths each gate events into a per-registry queue while paused; resume drains the queue [...]
-  - Extend the watch-control binding to liaison processes (`pkg/test/setup.startLiaisonNode`) and add `helpers.SharedContext.LiaisonAddr` so cluster-only specs can pause the receiving liaison's own `SchemaRegistry`. The cluster barrier's `selfName` probe reads through that SR, so pausing it surfaces a laggard via the public `AwaitX` RPCs.
-  - Author  cluster-barrier integration specs (`test/cases/schema/barrier_cluster.go`): all four specs now pass end-to-end.  (`AwaitSchemaApplied`) and  (`AwaitSchemaDeleted`) pin the per-key contract that a paused receiving liaison surfaces a non-empty `laggards` list and that resume drains the queue so the barrier converges.  (`AwaitRevisionApplied`) and  (cross-barrier recovery) pin the global watermark contract: the barrier's `GetMaxModRevision` advances past the target revision afte [...]
-  - Expose `cluster.v1.NodeSchemaStatusService` on data-node gRPC ports. Decouple the registration in `banyand/queue/sub/server.go`'s `Serve()` so `fodc.v1.GroupLifecycleService` (liaison-only by design) and `NodeSchemaStatusService` (per-node by design) are gated independently: the new `queue.Server.SetNodeSchemaStatusRepo(metadata.Service)` setter wires the per-node service without dragging along the liaison-shaped `GroupLifecycleService`. Liaison startup (`pkg/cmdsetup/liaison.go`) ca [...]
-  - Repair the `GetMaxRevision` aggregation on the per-node `NodeSchemaStatusService` (`banyand/metadata/schema/property/node_status.go`). The previous implementation returned `min(schemaCache.notifiedModRevision, NodeRepoRegistry.LatestModRevision)`, but `LatestModRevision` aggregated per-service `schemaRepo` watermarks via `min` — and each `schemaRepo` only advances on events for its own catalog (`pkg/schema/init.go:72` filters by `g.Catalog`), so the min was perpetually pinned to the  [...]
-  - Fix the `notifiedModRevision` watermark advancement in `SchemaRegistry.processInitialResourceFromProperty`, `handleWatchEvent` (DELETE branch), and `handleDeletion`. Previously `AdvanceNotified` was gated on `cache.Update` / `cache.Delete` returning true, but those methods compare `latestUpdateAt` (property timestamp) while the watermark tracks `modRevision` (etcd revision). When the property timestamp is stale (e.g. a no-op Update that doesn't change the measure spec), the cache rej [...]
-  - Fix the `modRevision` contract on no-op Update RPCs (`MeasureRegistryService.Update`, etc.). Previously `updateResource` detected unchanged content via `CheckerMap` and short-circuited without writing to the property store, but the caller had already fabricated `modRevision = time.Now().UnixNano()` and returned it. The returned revision never appeared in the property watch stream, so `AwaitRevisionApplied(R)` would hang. `updateResource` now returns `(int64, error)` — the existing pr [...]
-  - Add end-to-end observability for liaison internal queue pipelines with per-topic metrics for queue_sub and queue_pub, along with Grafana panels and troubleshooting docs.
-  - Introduce the migration tool with `copy`, `verify`, and `analyze` subcommands for both measure and stream data.
+- Schema consistency (Phase 2): add cluster-wide barriers that fan out `AwaitRevisionApplied`, `AwaitSchemaApplied`, and `AwaitSchemaDeleted` across liaison and data nodes through `NodeSchemaStatusService`, while safely handling mixed-version, membership, and timeout cases.
+- Align schema barriers and write/query revision gates with each node's executor cache, including correct watermark advancement and no-op update revisions.
+- Add schema-barrier metrics, structured access logs, integration coverage, and a load-test harness.
+- Introduce a migration tool with `copy`, `verify`, and `analyze` subcommands for measure and stream data.
 - Support displaying a measure's indexed tags in the dump tool, resolved per part so peak memory is bounded by the part rather than a segment-wide series map.
-- Snapshot/backup and data inspection no longer reopen idle-closed segments, avoiding cold-segment nil-index panics and index lock-file churn.
+- Make segment lifecycle safe for snapshots, backups, and inspection: segments retain a dormant state while idle, and these operations no longer reopen closed segments or race idle cleanup.
 - Add opt-in vectorized measure query tracing over raw-frame distributed queries, including a trace envelope and fixed trace-label vocabulary.
-- Enhance segment lifecycle: `refCount` now counts only active users, decoupled from "open" (`index != nil`), adding a "dormant" state (open, `refCount == 0`). A `DecRef` to zero no longer closes a segment; idle reclaim and retention delete act only at `refCount == 0`, so an in-flight snapshot/inspect is no longer torn down mid-operation (fixing the cold-node nil-index panic and bluge lock churn) while idle segments still release their bluge writers.
 - Speed up GCS backup uploads: write each object and its checksum metadata in one request, dropping the per-object `Update` round-trip.
 - Lifecycle migration now archives rows whose measure/stream schema was deleted from the registry, instead of aborting the group.
 - Canopy M3 metadata CRUD: add IndexRule + IndexRuleBinding create/edit/delete forms with admin-only edit/delete, pure `validate*()` module covering 8 input groups (group, stream, measure, trace, index rule, index rule binding, interval, lifecycle stages), `useFocusTrap` + `useDirtyGuard` modal hooks, live `m3-indexrule` e2e (CRUD round-trip + MF2 server-authority negative), and opt-in `m3-handoff` e2e + zero-dep static server for the design-bundle comparison.
-- Throttle lifecycle migration under receiver memory pressure so it no longer OOM-kills the destination data node, via three always-on mechanisms: per-chunk `SYNC_STATUS_SERVER_BUSY` load-shed (the sender backs off and retries); a bounded memory wait before introducing an external series-index segment (`{measure,stream,trace}-lifecycle-receive-mem-wait-timeout`, default 5m); and sender-side bounded-backoff retry of transient send failures — target node restarting/`no nodes available`, di [...]
-- Cut the lifecycle external series-index deduplicator's memory from ~1.8 GB to near-zero per segment by bumping bluge to a memory-efficient implementation: duplicates are now resolved via `_id` term-dictionary point lookups (`Dictionary.Contains`) instead of decompressing and retaining every segment's stored fields, removing the dominant heap source during external-segment introduction. No public API change and identical dedup semantics (verified by a differential test); this is the roo [...]
-- Add the storage-node in-merge trace retention filter (`PIPELINE_EVENT_MERGE`): a per-group sampler chain evaluates traces at merge time and drops non-retained traces from both the core part and the coupled sidx (`sidx.Merge` gains a keep predicate; a nil predicate restores the wholesale merge). Chain execution fails open on panic/error/length-mismatch/timeout, with a consecutive-timeout circuit breaker. Samplers are native Go plugins (`.so`) loaded at startup from `--trace-pipeline-con [...]
-- Register/update/remove in-merge trace retention samplers at runtime without restarting the data node, replacing the static `--trace-pipeline-config` boot loader (now removed). The sampler pipeline config is carried on `common.v1.Group` (new optional `pipeline` field, with `TracePipelineConfig`/`SamplerPlugin` relocated from `pipeline/v1` into the leaf `common/v1` to avoid a `common`→`pipeline` import cycle) and delivered through the existing `KindGroup` schema watch; the data node rebu [...]
+- Prevent lifecycle migration receiver OOMs with memory-efficient external series-index deduplication, receiver load shedding and bounded memory waits, and sender retries for transient failures; incomplete parts resume in the next cycle.
+- Add a storage-node in-merge trace-retention filter (`PIPELINE_EVENT_MERGE`) that evaluates per-group sampler chains and safely drops non-retained traces from core and secondary-index parts. Configure samplers dynamically through each group's pipeline, with bounded staging, fail-open error handling, and runtime register, update, and removal support.
 - Add FODC memory-pressure pprof capture: agent grabs heap/goroutine profiles when RSS nears the cgroup limit, served via the proxy.
 - Add fuzzy resource search to the From row: type a name and jump to any measure / stream / trace / Top-N across every group, with contiguous-run + word-boundary scoring, highlighted matches, arrow/Enter/Escape navigation, and click-outside dismiss.
 - Add bounded plugin telemetry (meter + logger) to the trace-pipeline sampler SDK. A sampler plugin opts in by implementing `sdk.HostAware`; the engine calls `UseHost(h)` once per group before the first `Decide`, delivering a scoped `sdk.Host` the plugin may cache and use in `Decide`. The host enforces all resource limits: metric names are forced under the `banyandb_trace_pipeline_plugin_` prefix; `{group, plugin_name}` labels are always present; per-plugin cardinality is capped at 100 s [...]
@@ -87,25 +62,22 @@ Release Notes.
 - Remove the `latencystatussampler` first-party plugin. It projected `duration` and `status`: `status` exists in neither shipped trace schema and `duration` only in Zipkin (in µs, so a `thresholdMs` there was off by 1000x), so it could never match live data and — failing open — kept every trace it was given. It shipped in the `-plugins-carrier` image beside the two real samplers, leaving three first-party plugins of which one silently retained everything. The near-identical copy under `t [...]
 - Add a Claude/Codex plugin packaging the BanyanDB MCP server with a parse-only `validate_bydbql` tool (backed by the `mcp/tools/bydbql-parse` Go validator) and a BydbQL skill for read-only natural-language-to-BydbQL generation over STREAM/MEASURE/TRACE/PROPERTY resources.
 - Introduce positional parameter binding (`?` placeholders) into BydbQL to eliminate QL injection.
-- Add reusable BydbQL binding: Prepare a query once, then Bind it many times without re-parsing or mutating the template. The liaison caches prepared statements on the gRPC query path (LRU bounded by entry count and bytes, on by default; `--bydbql-prepared-cache-size`/`--bydbql-prepared-cache-max-bytes`, `bydbql_prepared_cache_*` metrics) so repeated templates skip parsing. To pinpoint un-cacheable and slow queries without high-cardinality labels, the query access log tags each entry by  [...]
-- Expire BydbQL top-K entries not seen within their TTL (`--bydbql-topk-slow-ttl` / `--bydbql-topk-reparse-ttl`, default `24h`), and log `last_seen` / `max_latency_at`.
-- Add FODC schema consistency check comparing registry, cache, and runtime fingerprints across all nodes via lifecycle inspect.
-- Log the latest slow BydbQL query's bound parameters as `last_params`, redacted per `--bydbql-topk-param-mode` (default `fingerprint`).
+- Add reusable BydbQL binding and prepared-statement caching on the gRPC query path, with bounded cache and top-K retention, cache and slow-query observability, and redacted bound parameters in the latest slow-query log.
+- Add an FODC schema consistency check comparing registry, cache, and runtime fingerprints across all nodes via lifecycle inspection.
 
 ### Bug Fixes
 
-- Fix flaky trace snapshot-fallback tests (`TestTraceInitTSTableDeletesMultipleFailedSnapshotsOnFallback`, `TestTraceTolerantLoaderFallbackToOlderSnapshot`) that raced the async snapshot flush: they gated on the part directory, which is created at the start of the flush, so `Close()` could run before the snapshot manifest was persisted and observe zero `.snp` files. The tests now wait for the persisted `.snp` file itself, matching the measure/stream sibling tests.
+- Stabilize measure, stream, and trace snapshot tests by waiting for persisted manifests and all flushed memory parts rather than directory creation or the latest snapshot creator, eliminating asynchronous-flush and fallback races.
 - Prevent merge-time trace sampling from dropping fragments when the same trace ID may remain in unselected parts. Provisional drops are now checked against time bounds and
   trace-ID Bloom filters in candidate outside parts whose time bounds intersect the trace's max-fragment-gap-expanded range, revalidated before publication, and retained on
   uncertainty; the default merge grace is two hours.
-- FODC: expose node labels reliably and without collisions. The proxy overlaid each agent's node labels onto every metric under their raw keys, so a data node's `type=hot/warm/cold` tier label overwrote the merge metrics' own intrinsic `type="file"/"mem"` dimension on any node whose agent had resolved its labels. Two fixes: (1) the agent now stamps its node labels **live** from the cluster collector (which keeps resolving after startup) under a `node_` prefix per metric — so the tier rel [...]
+- FODC: namespace dynamic node labels as `node_*`, continuously resolve node roles and labels, and prevent oversized metric uploads and ghost `ROLE_UNSPECIFIED` series.
 - Close BanyanDB merge write-path durability gap that allowed torn parts to be created by a crash between data write and metadata commit. Metadata files (`metadata.json` for trace/measure/stream, `manifest.json` for sidx, plus `traceID.filter` and `tag.type`) now go through a new `WriteAtomic` (write-tmp + fsync + rename + fsync-dir) sequence; data writers (`seqWriter.Close`, `localFileSystem.Write`) now propagate fdatasync errors instead of silently dropping them. `mustOpenFilePart` / ` [...]
 - Fix bydbctl command tests using global stdout capture, which caused race-enabled runs to corrupt captured command output.
-- Use `topic` instead of `session_id` as the Prometheus label on liaison `queue_sub` chunk-ordering counters to avoid unbounded metric cardinality.
 - Fix flaky trace query filtering caused by non-deterministic sidx tag ordering and add consistency checks for integration query cases.
 - Fix index-mode measure queries returning documents outside the requested time range when a widened segment overlaps the query window.
 - MCP: Add validation for properties and harden the mcp server.
-- Fix property schema client connection not stable after data node restarted.
+- Make property-schema client connections recover after data-node restarts and add periodic health checks.
 - Fix flaky on-disk integration tests caused by Ginkgo v2 random container shuffling closing gRPC connections prematurely.
 - Fix snapshot error when there is no data in a segment.
 - ui: fix query editor refresh/reset behavior and BydbQL keyword highlighting.
@@ -123,13 +95,11 @@ Release Notes.
 - Fix `FileSystemError` not satisfying `errors.Is(err, io/fs.ErrNotExist)`, which prevented the segment controller from cleaning up half-born segment directories and left groups in a permanent zombie state after a crash or partial sync.
 - Fix lifecycle migration panic when a stream shard's snapshot has no element index (`idx/`) directory.
 - Avoid FODC lifecycle inspection failing on busy data nodes by raising the per-broadcast `CollectDataInfo` / `CollectLiaisonInfo` deadline from 5s to 30s and parallelizing per-group inspection in the cluster-internal `InspectAll`.
-- Fix flaky `file_snapshot` subtest in measure/stream/trace by waiting until every introduced mem part has been flushed to disk, instead of only checking the latest snapshot creator.
 - Fix deadlock when fodc-agent reconnects to fodc-proxy after a pod rotation.
 - Fix flaky `TestCollectWithPartialClosedSegments` by raising `SegmentIdleTimeout` so wall-clock variance on slow CI does not mark still-open segments as idle.
 - Fix FODC lifecycle cache poisoning where transient `InspectAll` failures were cached for 10 minutes and masked liaison recovery; raise FODC agent and proxy timeouts from 10s to 40s.
 - Fix FODC `/cluster/lifecycle` dropping zero-valued group fields (e.g. `replicas=0`, `close=false`) under `encoding/json` + `omitempty`; switch to `protojson` so all fields are emitted (nil nested messages serialize as `null`).
 - Fix trace `block_writer` panic on out-of-order timestamps within the same traceID, which dropped one trace-write batch per panic in multi-agent SkyWalking deployments. Spans of a single trace originate from independently-clocked services, and trace storage is organized by traceID rather than timestamp, so per-traceID timestamp monotonicity is not a writer invariant.
-- Fix nil-pointer panic on cold-tier data nodes when FODC `InspectAll` raced with idle-segment cleanup.
 - Add `GroupLifecycleInfo.errors` to surface per-group collection failures from FODC `InspectAll` instead of silently dropping the affected node entry.
 - Fix `CollectDataInfo` and `CollectLiaisonInfo` not handling `CATALOG_PROPERTY` groups.
 - Fix lifecycle migration where the receiving node could create segments shorter than the configured `SegmentInterval`.
@@ -140,9 +110,7 @@ Release Notes.
 - Fix trace query identity-tag projection: when `trace_id`/`span_id` are explicitly projected, reconstruct them from span identity at response build time instead of requesting them as stored tags, and preserve tag order with null-filled per-span value alignment in the distributed trace result iterator.
 - Fix measure, stream, and trace queries returning data from segments already expired by the TTL. Retention removes a segment only on its next scheduled run, so a fully expired segment can linger on disk and keep serving TTL-expired data; queries now skip segments whose whole time range is past the retention deadline, matching retention's own removal condition.
 - Trace storage metrics now expose the `storage` sub-scope, matching the `stream_storage_*` naming. The `StorageMetricsFactory` for trace switched from the root `trace` scope to `trace.storage`, so per-segment inverted-index metrics (`inverted_index_total_updates`, `inverted_index_total_doc_count`, `inverted_index_total_term_searchers_started`) are now emitted as `banyandb_trace_storage_*` instead of `banyandb_trace_*`, aligning the dashboard query names. Other trace metrics (`trace_tst_ [...]
-- Fix flaky measure snapshot tests that gated on the part directory appearing in `tab/` as the flush-completion signal. That directory is created by the first line of `memPart.mustFlush`, before the mem→file introduction reaches the in-memory snapshot and before the `.snp` manifest is persisted, so under `-race`/CI load `TakeFileSnapshot` could observe only mem parts and `Close` could drop the in-flight flush; gate on the persisted `.snp` manifest instead.
 - Fix FODC proxy corrupting Prometheus metric types. The agent dropped the `# TYPE` line while parsing banyandb `/metrics`, the `StreamMetrics` proto carried no type field, and the proxy guessed the type from a name-suffix heuristic — downgrading counters to gauge, mislabeling `_count`-suffixed counters as histograms, and splitting summaries into two conflicting `# TYPE` lines. Capture the type with the Prometheus `expfmt` parser, store it in the flight recorder, thread it through a new  [...]
-- Fix FODC agent labeling metrics with `node_role="ROLE_UNSPECIFIED"`. The agent resolved the node role exactly once at startup via a single `GetCurrentNode` poll whose endpoint retries spanned only ~1s; when the sibling lifecycle/banyandb gRPC server was not yet listening (`connect: cannot assign requested address`) the role fell back to `ROLE_UNSPECIFIED` permanently, so most nodes never reported their real `ROLE_DATA`/`ROLE_LIAISON`. Retry the initial node-role resolution with exponen [...]
 - Fix lifecycle row-replay OOM on large measure parts by streaming the dump reader, pooling size-classed marshal buffers, and bounding in-flight batch bytes (default 32 MiB); peak heap drops ~1.5 GB→~296 MB.
 - Consolidate lifecycle migration report errors into a single flat list of structured, stage-aware entries.
 - Fix backup container OOM from overlapping scheduled runs; serialize runs and upload small snapshot files concurrently.
@@ -160,7 +128,6 @@ Release Notes.
 - Build the plugin `.so` with `-tags slim` to match the host binary's ABI. Without this, `pkg/pool` has build-tag-dependent code (`tracker.go` vs `tracker_stub.go`) and the plugin-sidecar E2E failed with "plugin was built with a different version of package `pkg/pool`".
 - Make the trace merge loop survive a persistently failing merge instead of destroying the node. A production cluster wedged when a few unreadable parts made every merge selection fail: each failed attempt leaked its partially-written output directory (the block writer creates it before the merge runs), which exhausted the volume's inodes at 8% byte usage; the resulting disk-full panic then killed the merge lane workers, which are recovered but never respawned, so the dispatcher blocked  [...]
 - Report panics recovered by the gRPC servers through `panicdiag`, so they reach `banyandb_panic_total{component}`, the crash reporters and the artifact writer. Each server (`queue/sub`, `liaison/grpc`, `property/gossip`, `metadata/schema/schemaserver`) hand-rolled a recovery handler that only logged and returned `codes.Internal`, bypassing every panicdiag signal: during a production incident one data node logged 737 recovered request-path panics while `banyandb_panic_total` stayed at 4  [...]
-- Fix the flaky stream snapshot test, which gated on the part directory appearing in `tab/` as the flush-completion signal. That directory is created by the first line of `memPart.mustFlush`, before the mem->file introduction reaches the snapshot and before the `.snp` manifest is persisted, so `TakeFileSnapshot` could run mid-flush and copy no part at all (and read a half-written manifest). It now waits for the persisted manifest, the same fix already applied to the measure snapshot test [...]
 - Fix the measure schema-change merge test, which was time-of-day dependent rather than flaky. It wrote its two batches at `now-2h` and `now-1h` and waited for the part count to drop, but `sw_metric` uses a 1-day segment aligned to local midnight and a merge only ever combines parts within one segment, so any run starting within two hours of midnight split the batches across yesterday's and today's segments and the count could never drop. The budget had been raised twice (10x, then 20x)  [...]
 - Fix the flaky distributed schema clamp test: it queried once immediately after `AwaitRevision`/`AwaitApplied`, which confirm the schema cache but not that the data node has loaded the group's query topology, so the query could still fail with "group not found". It now retries until the query resolves, the same treatment the multi-group spec in the same file already had.
 - Stop the TopN test overriding the configured eventually timeout. It passed `flags.EventuallyTimeout` and then chained `WithTimeout(10s)`, which takes precedence, so the spec always ran on a 10s budget even though CI builds the integration suites with `-X ...flags.eventuallyTimeout=30s` to give slow runners room. It was the only case in `test/cases` opting out.
@@ -173,523 +140,10 @@ Release Notes.
 
 ### Chores
 
-- Upgrade Go and npm dependencies including etcd to v3.6.10, OpenTelemetry to v1.43.0, AWS SDK, and Google Cloud libraries.
-- Regenerate expired TLS test certificate with 100-year validity.
-- Regenerate the shared self-signed TLS test certificate, which expired at `Aug 10 06:54:23 2026 GMT` and broke every PR: the standalone TLS integration specs, the DNS discovery TLS specs, and the bydbctl health-check specs all load it. Same subject and SANs (`localhost`, `example.com`, `www.example.com`), now valid for 100 years. The earlier regeneration covered `banyand/metadata/schema/property/testdata/certs` but missed this pair, which is copied into both `test/integration/standalone [...]
+- Upgrade Go, npm, OpenTelemetry, AWS SDK, and Google Cloud dependencies.
+- Regenerate expired TLS test certificates with 100-year validity.
 - Set Ginkgo `--repeat` to 0 in the flaky-test workflow so the hourly run completes within the 50-minute timeout.
 - Refactor the dump tool into a reusable `banyand/dump` parser library.
 - Strip macOS AppleDouble (`._*`) and `__MACOSX/` metadata from every release tarball (src, banyand, bydbctl, fodc-agent, fodc-proxy) so downstream users running `make generate` from a downloaded source tarball no longer hit "invalid control character" errors when `buf generate` walks the resource-fork files; export `COPYFILE_DISABLE=1` and filter `._*` files at the source.
 - Bump Go and ui/mcp dependencies to clear Dependabot advisories: `golang.org/x/net` v0.52.0→v0.56.0 (CVE-2026-25680), `opencontainers/runc` v1.3.3→v1.3.6 (CVE-2026-41579). Refresh mcp/ui lockfiles and license attribution.
-- Defer the sw-trace-sampler tag-array decode to first use: 60% per-trace cost reduction on the realistic-mix path (`duration+err+tag` 2,710 → 1,015 ns/trace, 2,380 → 664 allocs/op on the SkyWalking schema).
-- Alias decoded entry strings via `pkg/convert.BytesToString`: 408 → 1 alloc/op on SkyWalking tag paths and 1,536 → 1 on Zipkin tag paths (the realistic 3-row × 8-entry trace); 24-40% ns/trace reduction on tag-using paths. Also fixes a pre-existing nil-deref in `keepTrace` when the array column is absent.
-- Reject non-matching tag entries in the sampler's rule matcher before the string comparison. Most entries belong to some other tag, so the common outcome is rejection; a length and first-byte check now precedes `strings.HasPrefix`, and rules naming the array column itself take a branch with no prefix handling at all. Tag-rule decisions drop a further 18% and full-scenario decisions 12-27%.
-- Read the trace sampler's scalar tag columns without materializing the SDK's decoded value. `sdk.Value` is 112 bytes and `TagColumn.At` returns it by value, so reading the duration, start-time and error columns per row cost more in `runtime.duffcopy` than in decoding. The sampler now reads those 8-byte cells directly, gated narrowly on the column's exact type so a malformed cell in a wrong-typed column still fails open with the same error. Duration-envelope decisions drop 80% (`duration [...]
-- Decode escape-free trace tag rows without copying them. The sw/zipkin trace sampler copied every tag row before decoding, because the var-array decode rewrites its input in place and the engine hands the same batch to every link of a sampler chain. That copy is only needed for a row that actually contains an escape, so rows without one now decode straight from the block with the entries aliasing it — safe because `assembleTraceBlock` already deep-copies each value into block-owned stor [...]
-- Precompute each tag rule's `"tagKey="` entry prefix at admission instead of rebuilding it on every `matchEntries` call (once per rule per trace).
-- Add an escape-free fast path to `vararray.UnmarshalVarArray`, the var-array decoder shared by sidx, trace query, index filtering, the dump tool and the sampler SDK. An entry containing no escape byte is located with two `bytes.IndexByte` scans instead of a per-byte loop, dropping that case from O(len) to O(1) (-91% at a 256-byte payload) and leaving the buffer untouched. An entry that does carry an escape resumes the in-place loop at the first escape rather than restarting, since the b [...]
-
-## 0.10.0
-
-### Features
-
-- Remove Bloom filter for dictionary-encoded tags.
-- Implement BanyanDB MCP.
-- Support deleting non-entity tags when updating the schema.
-- Remove check requiring tags in criteria to be present in projection.
-- Add sorted query support for the Property.
-- Update bydbQL to add sorted query support for the Property.
-- Remove the windows arch for binary and docker image.
-- Support writing data with specifications.
-- Persist series metadata in liaison queue for measure, stream and trace models.
-- Update the dump tool to support analyzing the parts with smeta files.
-- Add replication integration test for measure.
-- Activate the property repair mechanism by default.
-- Add snapshot time retention policy to ensure the snapshot only can be deleted after the configured minimum age(time).
-- **Breaking Change**: Change the data storage path structure for property model:
-  - From: `<data-dir>/property/data/shard-<id>/...`
-  - To: `<data-dir>/property/data/<group>/shard-<id>/...`
-- Add a generic snapshot coordination package for atomic snapshot transitions across trace and sidx.
-- Support map-reduce aggregation for measure queries: map phase (partial aggregation on data nodes) and reduce phase (final aggregation on liaison).
-- Add eBPF-based KTM I/O monitor for FODC agent.
-- Support relative paths in configuration.
-- Support 'none' node discovery and make it the default.
-- Support server-side element ID generation for stream writes when clients omit element_id.
-- Implement entire group deletion.
-
-### Bug Fixes
-
-- Fix the wrong retention setting of each measure/stream/trace.
-- Fix server got panic when create/update property with high dist usage.
-- Fix incorrect key range update in sidx part metadata.
-- Fix panic in measure block merger when merging blocks with overlapping timestamps.
-- Fix unsupported empty string tag bug.
-- Fix duplicate elements in stream query results by implementing element ID-based deduplication across scan, merge, and result building stages.
-- Fix data written to the wrong shard and related stream queries.
-- Fix the lifecycle panic when the trace has no sidx.
-- Fix panic in sidx merge and flush operations when part counts don't match expectations.
-- Fix trace queries with range conditions on the same tag (e.g., duration) combined with ORDER BY by deduplicating tag names when merging logical expression branches.
-- Fix sidx tag filter range check returning inverted skip decision and use correct int64 encoding for block min/max.
-- Ignore take snapshot when no data.
-- Fix measure standalone write handler resetting accumulated groups on error, which dropped all successfully processed events in the batch.
-- Fix memory part reference leak in mustAddMemPart when tsTable loop closes.
-- Fix memory part leak in syncPartContext Close and prevent double-release in FinishSync.
-- Fix segment reference leaks in measure/stream/trace queries and ensure chunked sync sessions close part contexts correctly.
-- Fix duplicate query execution in distributed measure Agg+TopN queries by enabling push-down aggregation, removing the wasteful double-query pattern.
-- Fix nil pointer panic in segment collectMetrics during shutdown.
-- Fix entity tag handling in trace filter to prevent TagIdx index mismatch when filtering with both entity and non-entity tags.
-- Fix OOM issue cause during migration when a group contains a large amount of data.
-
-### Document
-
-- Add read write benchmark document for 0.9.0 release.
-- Add design of KTM.
-- Add FODC overview doc.
-- Remove Java client doc, and recreate client APIs docs.
-- Add common issue documentation.
-
-### Chores
-
-- Upgrade Node.js support from 20.12 to 24.6.0, and align CI, license checks, and documentation
-- Add Claude Code skill for vendor dependency updates.
-- Upgrade Go vendor dependencies and sync BPF2GO_VERSION with cilium/ebpf library.
-
-## 0.9.0
-
-### Features
-
-- Add sharding_key for TopNAggregation source measure
-- API: Update the data matching rule from the node selector to the stage name.
-- Add dynamical TLS load for the gRPC and HTTP server.
-- Implement multiple groups query in one request.
-- Replica: Replace Any with []byte Between Liaison and Data Nodes
-- Replica: Support configurable replica count on Group.
-- Replica: Move the TopN pre-calculation flow from the Data Node to the Liaison Node.
-- Add a wait and retry to write handlers to avoid the local metadata cache being loaded.
-- Implement primary block cache for measure.
-- Implement versioning properties and replace physical deletion with the tombstone mechanism for the property database.
-- Implement skipping index for stream.
-- Add Load Balancer Feature to Liaison. 
-- Implement fadvise for large files to prevent page cache pollution.
-- Data Model: Introduce the `Trace` data model to store the trace/span data.
-- Support dictionary encoding for low cardinality columns.
-- Push down aggregation for topN query.
-- Push down min/max aggregation to data nodes
-- Introduce write queue mechanism in liaison nodes to efficiently synchronize stream and measure partition folders, improving write throughput and consistency
-- Add trace module metadata management.
-- Add chunked data sync to improve memory efficiency and performance during data transfer operations, supporting configurable chunk sizes, retry mechanisms, and out-of-order handling for both measure and stream services.
-- Implement comprehensive migration system for both measure and stream data with file-based approach and enhanced progress tracking
-- Backup/Restore: Add support for AWS S3, Google Cloud Storage (GCS), and Azure Blob Storage as remote targets for backup and restore operations.
-- Improve TopN processing by adding "source" tag to track node-specific data, enhancing data handling across distributed nodes
-- Implement Login with Username/Password authentication in BanyanDB
-- Enhance flusher and introducer loops to support merging operations, improving efficiency by eliminating the need for a separate merge loop and optimizing data handling process during flushing and merging
-- Enhance stream synchronization with configurable sync interval - Allows customization of synchronization timing for better performance tuning
-- Refactor flusher and introducer loops to support conditional merging - Optimizes data processing by adding conditional logic to merge operations
-- New storage engine for trace:
-  - Data ingestion and retrieval.
-  - Flush memory data to disk.
-  - Merge memory data and disk data.
-- Enhance access log functionality with sampling option.
-- Implement a resilient publisher with circuit breaker and retry logic with exponential backoff.
-- Optimize gRPC message size limits: increase server max receive message size to 16MB and client max receive message size to 32MB for better handling of large time-series data blocks.
-- Add query access log support for stream, measure, trace, and property services to capture and log all query requests for monitoring and debugging purposes.
-- Implement comprehensive version compatibility checking for both regular data transmission and chunked sync operations, ensuring proper API version and file format version validation with detailed error reporting and graceful handling of version mismatches.
-- **Breaking Change**: Rename disk usage configuration flags and implement forced retention cleanup:
-  - `*-max-disk-usage-percent` → `*-retention-high-watermark` (measure, stream, trace, property services)
-  - Add new retention configuration flags: `*-retention-low-watermark`, `*-retention-check-interval`, `*-retention-cooldown`
-  - Implement disk monitor with forced retention cleanup for data/standalone servers
-  - Add comprehensive disk management documentation with configuration guides and troubleshooting
-- Implement cluster mode for trace.
-- Implement Trace views.
-- Use Fetch request to instead of axios request and remove axios.
-- Implement Trace Tree for debug mode.
-- Implement bydbQL.
-- UI: Implement the Query Page for BydbQL.
-- Refactor router for better usability.
-- Implement the handoff queue for Trace.
-- Add dump command-line tool to parse and display trace part data with support for CSV export and human-readable timestamp formatting.
-- Implement backoff retry mechanism for sending queue failures.
-- Implement memory load shedding and dynamic gRPC buffer sizing for liaison server to prevent OOM errors under high-throughput write traffic.
-- Add stream dump command to parse and display stream shard data with support for CSV export, filtering, and projection.
-
-### Bug Fixes
-
-- Fix the deadlock issue when loading a closed segment.
-- Fix the issue that the etcd watcher gets the historical node registration events.
-- Fix the crash when collecting the metrics from a closed segment.
-- Fix topN parsing panic when the criteria is set.
-- Remove the indexed_only field in TagSpec.
-- Fix returning empty result when using IN operatior on the array type tags.
-- Fix memory leaks and OOM issues in streaming processing by implementing deduplication logic in priority queues and improving sliding window memory management.
-- Fix etcd prefix matching any key that starts with this prefix.
-- Fix the sorting timestamps issue of the measure model when there are more than one segment.
-- Fix comparison issues in TopN test cases
-
-### Document
-
-- Introduce AI_CODING_GUIDELINES.md to provide guidelines for using AI assistants (like Claude, Cursor, GitHub Copilot) in development, ensuring generated code follows project standards around variable shadowing, imports, error handling, code style and documentation
-
-## 0.8.0
-
-### Features
-
-- Add the `bydbctl analyze series` command to analyze the series data.
-- Index: Remove sortable field from the stored field. If a field is sortable only, it won't be stored.
-- Index: Support InsertIfAbsent functionality which ensures documents are only inserted if their docIDs are not already present in the current index. There is a exception for the documents with extra index fields more than the entity's index fields.
-- Measure: Introduce "index_mode" to save data exclusively in the series index, ideal for non-timeseries measures.
-- Index: Use numeric index type to support Int and Float
-- TopN: Group top n pre-calculation result by the group key in the new introduced `_top_n_result` measure, which is used to store the pre-calculation result.
-- Index Mode: Index `measure_name` and `tags` in `entity` to improve the query performance.
-- Encoding: Improve the performance of encoding and decoding the variable-length int64.
-- Index: Add a cache to improve the performance of the series index write.
-- Read cpu quota and limit from the cgroup file system to set gomaxprocs.
-- Property: Add native storage layer for property.
-- Add the max disk usage threshold for the `Measure`, `Stream`, and `Property` to control the disk usage.
-- Add the "api version" service to gRPC and HTTP server.
-- Metadata: Wait for the existing registration to be removed before registering the node.
-- Stream: Introduce the batch scan to improve the performance of the query and limit the memory usage.
-- Add memory protector to protect the memory usage of the system. It will limit the memory usage of the querying.
-- Metadata: Introduce the periodic sync to sync the metadata from the etcd to the local cache in case of the loss of the events.
-- Test: Add the e2e test for zipkin.
-- Test: Limit the CPU and memory usage of the e2e test.
-- Add taking the snapshot of data files.
-- Add backup command line tool to backup the data files.
-- Add restore command line tool to restore the data files.
-- Add concurrent barrier to partition merge to improve the performance of the partition merge.
-- Improve the write performance.
-- Add node labels to classify the nodes.
-- Add lifecycle management for the node.
-- Property: Introduce the schema style to the property.
-- Add time range parameters to stream index filter.
-- UI: Add the `stages` to groups.
-- Add time range return value from stream local index filter.
-- Deduplicate the documents on building the series index.
-
-### Bug Fixes
-
-- Fix the bug that TopN processing item leak. The item can not be updated but as a new item.
-- Resolve data race in Stats methods of the inverted index.
-- Fix the bug when adding new tags or fields to the measure, the querying crashes or returns wrong results.
-- Fix the bug that adding new tags to the stream, the querying crashes or returns wrong results.
-- UI: Polish Index Rule Binding Page and Index Page.
-- Fix: View configuration on Property page.
-- UI: Add `indexMode` to display on the measure page.
-- UI: Refactor Groups Tree to optimize style and fix bugs.
-- UI: Add `NoSort` Field to IndexRule page.
-- Metadata: Fix the bug that the cache load nil value that is the unknown index rule on the index rule binding.
-- Queue: Fix the bug that the client remove a registered node in the eviction list. The node is controlled by the recovery loop, doesn't need to be removed in the failover process.
-- UI: Add prettier to enforce a consistent style by parsing code.
-- Parse string and int array in the query result table.
-- Fix the bug that fails to update `Group` Schema's ResourceOpts.
-- UI: Implement TopNAggregation data query page.
-- UI: Update BanyanDB UI to Integrate New Property Query API.
-- UI: Fix the Stream List.
-- Fix the oom issue when loading too many unnecessary parts into memory.
-- Fix data race between flusher/merger/syncer senders and the introducer loop during shutdown, caught by the Go race detector under CI pressure.
-- bydbctl: Fix the bug that the bydbctl can't parse the absolute time flag.
-
-### Documentation
-
-- Improve the description of the memory in observability doc.
-- Update kubernetes install document to align the banyandb helm v0.3.0.
-- Add restrictions on updating schema.
-- Add docs for the new property storage.
-- Update quick start guide to use showcase instead of the old example.
-
-### Chores
-
-- Fix metrics system typo.
-- Bump up OAP in CI to 6d262cce62e156bd197177abb3640ea65bb2d38e.
-- Update cespare/xxhash to v2 version.
-- Bump up Go to 1.24.
-
-### CVEs
-
-- GO-2024-3321: Misuse of ServerConfig.PublicKeyCallback may cause authorization bypass in golang.org/x/crypto
-- GO-2024-3333: Non-linear parsing of case-insensitive content in golang.org/x/net/html
-
-## 0.7.0
-
-### File System Changes
-
-- Bump up the version of the file system to 1.1.0 which is not compatible with the previous version.
-- Move the series index into segment.
-- Swap the segment and the shard.
-- Move indexed values in a measure from data files to index files.
-- Merge elementIDs.bin and timestamps.bin into a single file.
-
-### Features
-
-- Check unregistered nodes in background.
-- Improve sorting performance of stream.
-- Add the measure query trace.
-- Assign a separate lookup table to each group in the maglev selector.
-- Convert the async local pipeline to a sync pipeline.
-- Add the stream query trace.
-- Add the topN query trace.
-- Introduce the round-robin selector to Liaison Node.
-- Optimize query performance of series index.
-- Add liaison, remote queue, storage(rotation), time-series tables, metadata cache and scheduler metrics.
-- Add HTTP health check endpoint for the data node.
-- Add slow query log for the distributed query and local query.
-- Support applying the index rule to the tag belonging to the entity.
-- Add search analyzer "url" which breaks test into tokens at any non-letter and non-digit character.
-- Introduce "match_option" to the "match" query.
-
-### Bugs
-
-- Fix the filtering of stream in descending order by timestamp.
-- Fix querying old data points when the data is in a newer part. A version column is introduced to each data point and stored in the timestamp file.
-- Fix the bug that duplicated data points from different data nodes are returned.
-- Fix the bug that the data node can't re-register to etcd when the connection is lost.
-- Fix memory leak in sorting the stream by the inverted index.
-- Fix the wrong array flags parsing in command line. The array flags should be parsed by "StringSlice" instead of "StringArray".
-- Fix a bug that the Stream module didn't support duplicated in index-based filtering and sorting
-- Fix the bug that segment's reference count is increased twice when the controller try to create an existing segment.
-- Fix a bug where a distributed query would return an empty result if the "limit" was set much lower than the "offset".
-- Fix duplicated measure data in a single part.
-- Fix several "sync.Pool" leak issues by adding a tracker to the pool.
-- Fix panic when removing a expired segment.
-- Fix panic when reading a disorder block of measure. This block's versions are not sorted in descending order.
-- Fix the bug that the etcd client doesn't reconnect when facing the context timeout in the startup phase.
-- Fix the bug that the long running query doesn't stop when the context is canceled.
-- Fix the bug that merge block with different tags or fields.
-- Fix the bug that the pending measure block is not released when a full block is merged.
-
-### Documentation
-
-- Introduce new doc menu structure.
-- Add installation on Docker and Kubernetes.
-- Add quick-start guide.
-- Add web-ui interacting guide.
-- Add bydbctl interacting guide.
-- Add cluster management guide.
-- Add operation related documents: configuration, troubleshooting, system, upgrade, and observability.
-
-### Chores
-
-- Bump up the version of infra e2e framework.
-- Separate the monolithic release package into two packages: banyand and bydbctl.
-- Separate the monolithic Docker image into two images: banyand and bydbctl.
-- Update CI to publish linux/amd64 and linux/arm64 Docker images.
-- Make the build system compiles the binary based on the platform which is running on.
-- Push "skywalking-banyandb:<tag>-testing" image for e2e and stress test. This image contains bydbctl to do a health check.
-- Set etcd-client log level to "error" and etcd-server log level to "warn".
-- Push "skywalking-banyandb:<tag>-slim" image for the production environment. This image doesn't contain bydbctl and Web UI.
-- Bump go to 1.23.
-
-## 0.6.1
-
-### Features
-
-- Limit the max pre-calculation result flush interval to 1 minute.
-- Use both datapoint timestamp and server time to trigger the flush of topN pre-calculation result.
-- Add benchmarks for stream filtering and sorting.
-- Improve filtering performance of stream.
-
-### Bugs
-
-- Fix the bug that topN query doesn't return when an error occurs.
-- Data race in the hot series index selection.
-- Remove SetSchema from measure cache which could change the schema in the cache.
-- Fix duplicated items in the query aggregation top-n list.
-- Fix non-"value" field in topN pre-calculation result measure is lack of data.
-- Encode escaped characters to int64 bytes to fix the malformed data.
-
-## 0.6.0
-
-### Features
-
-- Support etcd client authentication.
-- Implement Local file system.
-- Add health check command for bydbctl.
-- Implement Inverted Index for SeriesDatabase.
-- Remove Block Level from TSDB.
-- Remove primary index.
-- Measure column-based storage:
-  - Data ingestion and retrieval.
-  - Flush memory data to disk.
-  - Merge memory data and disk data.
-- Stream column-based storage:
-  - Data ingestion and retrieval.
-  - Flush memory data to disk.
-  - Merge memory data and disk data.
-- Add HTTP services to TopNAggregation operations.
-- Add preload for the TopN query of index.
-- Remove "TREE" index type. The "TREE" index type is merged into "INVERTED" index type.
-- Remove "Location" field on IndexRule. Currently, the location of index is in a segment.
-- Remove "BlockInterval" from Group. The block size is determined by the part.
-- Support querying multiple groups in one request.
-
-### Bugs
-
-- Fix the bug that property merge new tags failed.
-- Fix CPU Spike and Extended Duration in BanyanDB's etcd Watching Registration Process.
-- Fix panic when closing banyand.
-- Fix NPE when no index filter in the query.
-
-### Chores
-
-- Bump go to 1.22.
-- Bump node to 2.12.2.
-- Bump several tools.
-- Bump all dependencies of Go and Node.
-- Combine banyand and bydbctl Dockerfile.
-- Update readme for bydbctl
-- Introduce the go vulnerability check to "pre-push" task.
-
-## 0.5.0
-
-### Features
-
-- List all properties in a group.
-- Implement Write-ahead Logging
-- Document the clustering.
-- Support multiple roles for banyand server.
-- Support for recovery buffer using wal.
-- Register the node role to the metadata registry.
-- Implement the remote queue to spreading data to data nodes.
-- Implement the distributed query engine.
-- Add mod revision check to write requests.
-- Add TTL to the property.
-- Implement node selector (e.g. PickFirst Selector, Maglev Selector).
-- Unified the buffers separated in blocks to a single buffer in the shard.
-
-### Bugs
-
-- BanyanDB ui unable to load icon.
-- BanyanDB ui type error
-- Fix timer not released
-- BanyanDB ui misses fields when creating a group
-- Fix data duplicate writing
-- Syncing metadata change events from etcd instead of a local channel.
-- Fix parse environment variables error.
-- Fix console warnings in dev mod, and optimize `vite` configuration for proxy.
-
-### Chores
-
-- Bump several dependencies and tools.
-- Drop redundant "discovery" module from banyand. "metadata" module is enough to play the node and shard discovery role.
-
-## 0.4.0
-
-### Features
-
-- Add TSDB concept document.
-- [UI] Add YAML editor for inputting query criteria.
-- Refactor TopN to support `NULL` group while keeping seriesID from the source measure.
-- Add a sharded buffer to TSDB to replace Badger's memtable. Badger KV only provides SST.
-- Add a meter system to control the internal metrics.
-- Add multiple metrics for measuring the storage subsystem.
-- Refactor callback of TopNAggregation schema event to avoid deadlock and reload issue.
-- Fix max ModRevision computation with inclusion of `TopNAggregation`
-- Enhance meter performance
-- Reduce logger creation frequency
-- Add units to memory flags
-- Introduce TSTable to customize the block's structure
-- Add `/system` endpoint to the monitoring server that displays a list of nodes' system information.
-- Enhance the `liaison` module by implementing access logging.
-- Add the Istio scenario stress test based on the data generated by the integration access log.
-- Generalize the index's docID to uint64.
-- Remove redundant ID tag type.
-- Improve granularity of index in `measure` by leveling up from data point to series.
-- [UI] Add measure CRUD operations.
-- [UI] Add indexRule CRUD operations.
-- [UI] Add indexRuleBinding CRUD operations.
-
-### Bugs
-
-- Fix iterator leaks and ensure proper closure and introduce a closer to guarantee all iterators are closed
-- Fix resource corrupts caused by update indexRule operation
-- Set the maximum integer as the limit for aggregation or grouping operations when performing aggregation or grouping operations in a query plan.
-
-### Chores
-
-- Bump go to 1.20.
-- Set KV's minimum memtable size to 8MB
-- [docs] Fix docs crud examples error
-- Modified `TestGoVersion` to check for CPU architecture and Go Version
-- Bump node to 18.16
-
-## 0.3.1
-
-### Bugs
-
-- Fix the broken of schema chain.
-- Add a timeout to all go leaking checkers.
-
-### Chores
-
-- Bump golang.org/x/net from 0.2.0 to 0.7.0.
-
-## 0.3.0
-
-### Features
-
-- Support 64-bit float type.
-- Web Application.
-- Close components in tsdb gracefully.
-- Add TLS for the HTTP server.
-- Use the table builder to compress data.
-
-### Bugs
-
-- Open blocks concurrently.
-- Sync index writing and shard closing.
-- TimestampRange query throws an exception if no data in this time range.
-
-### Chores
-
-- Fixes issues related to leaked goroutines.
-- Add validations to APIs.
-
-For more details by referring to [milestone 0.3.0](https://github.com/apache/skywalking/issues?q=is%3Aissue+milestone%3A%22BanyanDB+-+0.3.0%22)
-
-## 0.2.0
-
-### Features
-
-- Command line tool: bydbctl.
-- Retention controller.
-- Full-text searching.
-- TopN aggregation.
-- Add RESTFul style APIs based on gRPC gateway.
-- Add "exists" endpoints to the schema registry.
-- Support tag-based CRUD of the property.
-- Support index-only tags.
-- Support logical operator(and & or) for the query.
-
-### Bugs
-
-- "metadata" syncing pipeline complains about an "unknown group".
-- "having" semantic inconsistency.
-- "tsdb" leaked goroutines.
-
-### Chores
-
-- "tsdb" structure optimization.
-  - Merge the primary index into the LSM-based index
-  - Remove term metadata.
-- Memory parameters optimization.
-- Bump go to 1.19.
-
-For more details by referring to [milestone 0.2.0](https://github.com/apache/skywalking/issues?q=is%3Aissue+milestone%3A%22BanyanDB+-+0.2.0%22)
-
-## 0.1.0
-
-### Features
-
-- BanyanD is the server of BanyanDB
-  - TSDB module. It provides the primary time series database with a key-value data module.
-  - Stream module. It implements the stream data model's writing.
-  - Measure module. It implements the measure data model's writing.
-  - Metadata module. It implements resource registering and property CRUD.
-  - Query module. It handles the querying requests of stream and measure.
-  - Liaison module. It's the gateway to other modules and provides access endpoints to clients.
-- gRPC based APIs
-- Document
-  - API reference
-  - Installation instrument
-  - Basic concepts
-- Testing
-  - UT
-  - E2E with Java Client and OAP
+- Optimize the SkyWalking and Zipkin trace samplers and shared var-array decoder through deferred decoding, zero-copy string and tag handling, early tag rejection, direct scalar reads, cached rule prefixes, and escape-free decoding paths.