You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "mrproliu (via GitHub)" <gi...@apache.org> on 2026/08/04 02:59:45 UTC
[PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
mrproliu opened a new pull request, #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244
## What
Adds an **FODC schema consistency check**: a cluster-wide, read-only health signal that detects when a node's schema cache or runtime view has drifted from the metadata registry.
For every group, the checker compares three fingerprints per object:
- **registry** โ the source of truth, read once from a schema-serving node
- **cache** โ what each node has cached
- **runtime** โ what each node is actually serving (materialized schema + index rules)
The verdict is exposed object-centrically on `GroupLifecycleService.InspectAll` and aggregated by fodc-proxy at `GET /cluster/lifecycle`.
## Why
Schema propagation is asynchronous; a stale cache or an un-applied index rule on a single node can silently corrupt reads. Until now there was no way to observe this from outside. This gives operators a steady-state "is every node serving the schema the registry thinks it should" signal.
## How it works
- **Fingerprinting** (`pkg/schema/consistency`): deterministic proto marshal + xxhash over each object plus its derived index-rule set.
- **Verdict** (`Checker`): registry vs. per-node cache/runtime, rolled up to `CONSISTENT` / `INCONSISTENT` / `UNKNOWN`. A divergence must survive **two consecutive rounds** before it is reported, filtering in-flight propagation and torn reads. An incomplete roster downgrades to `UNKNOWN` rather than passing a silent node as consistent.
- **Collection**: node evidence rides on the existing `CollectDataInfo`/`CollectLiaisonInfo` inspect path, folded in behind a new `include_schema_state` request flag. Only the FODC path sets it; the public `GroupRegistryService.Inspect` RPC honors the caller's flag (default `false`) and group-delete never sets it, so neither pays for the per-object fingerprinting or the extra payload.
- **Merge** (fodc-proxy): each agent reports its own verdict; the proxy keeps the **most severe** status and the **deduped union** of issues and per-object node fingerprints, so one healthy node can never mask another node's finding.
- **Error surfacing**: registry-build, node-roster, and liaison-collection failures are reported on `GroupLifecycleInfo.errors`, not just logged.
## Changes
| Area | Change |
| --- | --- |
| `api/proto/.../database/v1` | `ObjectSchemaState`, `include_schema_state` on inspect request |
| `api/proto/.../fodc/v1` | `SchemaConsistency` object-centric verdict on `GroupLifecycleInfo` |
| `pkg/schema/consistency` | fingerprint, derive, object sort, suppression-aware `Checker` |
| `banyand/queue/sub` | registry fingerprint build + per-group verdict wiring |
| `banyand/{measure,stream,trace}` | schema-state collection folded into the collectors |
| `fodc/proxy` | verdict merge + `/cluster/lifecycle` exposure |
| `test/e2e-v2/cases/fodc` | steady-state consistency guard |
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Fixes apache/skywalking#<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking-banyandb/blob/main/CHANGES.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on PR #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244#issuecomment-5199577979
Add `NodeSchemaStateService` in each liaison/data node to collect the cached schema. Then all the fingerprints are generated on the FODC `proxy/agent` side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244#issuecomment-5186594543
## ๐ซ Requested changes
### Move `xxhash` fingerprint compute out of `banyandb` core into a FODC sidecar agent
The current PR embeds fingerprint generation, suppression state, and registry fingerprint building into the data path (`banyand/queue/sub/schema_consistency.go`, `pkg/schema/collect_state.go`, the per-shard `metadata.go` files, the new `pkg/schema/consistency/` package, plus the `include_schema_state` flag on `Inspect`). This is observability logic in the data path. Before merge, please invert the design so the data path returns its existing protos and the FODC agent fingerprints on receive.
#### Why this blocks
**Decoupling of observability from the data path.** FODC's hash algorithm, suppression policy, and classification rules should be able to ship independently of `banyandb`. Today they're pinned.
**Footprint in `banyandb` is unnecessary.** ~150 LOC of fingerprint code in the data path plus 4 new proto fields on an existing RPC. None of it is needed if the agent owns the compute.
**Same wire integrity guarantees.** The trust boundary moves one hop; its posture is unchanged. There's no defensible reason to keep the compute inside `banyandb`.
#### Proposed architecture
```mermaid
sequenceDiagram
autonumber
actor Op as Operator / dashboard
participant FP as fodc-proxy
participant A as fodc-agent (sidecar)
participant L as Liaison (schema-serving)
participant DN as Data nodes
Op->>+FP: GET /cluster/lifecycle
FP->>+A: collect(group)
A->>+L: fetch registry view (existing RPC)
L-->>-A: registry proto
A->>A: registry_fp = xxhash(deterministic-marshal + derived rules)
par one server-stream per data node
A->>+DN: StreamGroupSchemaState(group)
DN-->>A: SchemaSnapshotEvent.object (N times)
DN-->>-A: SchemaSnapshotEvent.trailer{node, version, count, errors}
and
A->>+DNo: StreamGroupSchemaState(group)
DNo-->>-A: ...stream...
DNo-->>-A: trailer
end
A->>A: Checker (two-strike, streaks in agent)
A-->>-FP: SchemaConsistency{verdict, objects, issues}
Note over FP: merge ยท most severe + deduped union
FP-->>-Op: 200 OK ยท aggregated GroupLifecycleInfo
```
#### Required
The `pkg/schema/consistency/` package moves to the FODC agent binary (e.g. `fodc/agent/consistency/`), replacing the new package in `banyandb`.
`banyand/queue/sub/schema_consistency.go` is deleted. The registry fingerprint is built by the agent from the registry RPC response rather than inside queue/sub.
The per-shard `metadata.go` files for measure, stream, and trace revert to no fingerprint code.
`pkg/schema/collect_state.go` is deleted; collection lives in the agent now.
The `include_schema_state` flag on `GroupRegistryServiceInspectRequest` is removed.
The `ObjectSchemaState` field on `DataInfo` and `LiaisonInfo` is removed.
The per-shard `metadata_consistency_test.go` files (ร3) are deleted; equivalent coverage moves into `fodc/agent/consistency_test.go`.
#### Preserved (no change)
- `Checker` algorithm (two-strike suppression, classify).
- `xxhash` + `clearVolatile` fingerprint format.
- `ObjectConsistency` / `SchemaIssue` / `SchemaConsistency` wire shape exposed by the proxy.
- Proxy merge logic (most-severe + deduped union), including the `len(nodes) < expectedNodes โ UNKNOWN` rule from `pkg/schema/consistency/checker.go:144`.
- Existing per-catalog fingerprint identity semantics in `pkg/schema/consistency/fingerprint.go:92-107`.
Companion comment below has the streaming proto API the agent talks to โ they're a pair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244#issuecomment-5186595066
## ๐ซ Requested changes
### Replace `ObjectSchemaState` (uint64 fingerprints) with a streaming `NodeSchemaStateService` that returns proto bodies
The companion comment above moves fingerprint compute to the agent. To make that work, the data path must return each node's schema **proto body**, not uint64 fingerprints. This PR's `ObjectSchemaState` (uint64s) and `include_schema_state` flag must be removed; in their place, add `NodeSchemaStateService.StreamGroupSchemaState`, which streams the actual snapshot objects.
Before merge, please make this change.
#### Why this blocks
The current `ObjectSchemaState` is a dead end for the agent-side design. Once fingerprint code lives in the agent, shipping pre-hashed uint64s from `banyandb` is wasted work on the hot path.
`Inspect` cannot stream; this RPC must. At ~3 KB per schema object (proto body + bound index rules), a unary RPC hits gRPC's default 4 MB ceiling at ~1k objects and breaks past ~5k. Streaming with bounded per-message size sidesteps the limit and lets the agent fan out to N data nodes in parallel โ total wall-clock is the slowest single node, not the sum.
`SchemaInfo` returns names, not protos. This is the only clean way to get proto bodies without N+1 `GetSchema` calls.
#### Required: new RPC
```proto
service NodeSchemaStateService {
// StreamGroupSchemaState returns this node's full snapshot for one
// group. The server holds a snapshot lock for the duration of the
// stream so every yielded object comes from the same view; mid-stream
// watch events queue behind the stream's release.
//
// Empty stream (zero object events) is a normal termination โ the
// agent treats the absence of objects as "this node doesn't cache
// this group", not as an error.
rpc StreamGroupSchemaState(NodeSchemaStateRequest)
returns (stream SchemaSnapshotEvent);
}
message NodeSchemaStateRequest {
string group = 1;
// internal_version correlates the snapshot with what the agent
// asked for. Zero means "no version pinning, best-effort" โ the
// agent then treats the stream as torn-read suspect and lets the
// two-round suppression absorb it.
uint64 internal_version = 2;
}
// SchemaSnapshotEvent is the streaming element. Wire-extensible โ new
// event kinds add oneof variants without breaking older agents.
// Stream terminates either with a SchemaSnapshotTrailer (clean) or a
// non-EOF status (transport error).
message SchemaSnapshotEvent {
oneof event {
SchemaObjectSnapshot object = 1;
SchemaSnapshotTrailer trailer = 2;
}
}
// One schema object's body, generated from the server's snapshot.
// The server's snapshot_version is identical for every object event
// in a single stream; the agent rejects streams with version drift.
message SchemaObjectSnapshot {
uint64 snapshot_version = 1;
ObjectSnapshot object = 2;
}
// Final message of a successful stream. Always sent before EOF.
message SchemaSnapshotTrailer {
// node echoes the responding node so the agent can detect misrouted
// responses in mixed-version clusters (older nodes may field
// requests on behalf of newer ones).
string node = 1;
// snapshot_version mirrors the value tagged on every object event
// in this stream โ agent verifies first-event and trailer agree.
uint64 snapshot_version = 2;
// object_count is a sanity check the agent can verify against the
// number of objects it received. Mismatch โ torn stream, retry.
uint32 object_count = 3;
// errors are failures observed during snapshot collection. They
// flow up to GroupLifecycleInfo.errors but do NOT cause the stream
// to fail โ partial successes are still useful.
repeated string errors = 4;
}
// ObjectSnapshot is one schema object's body plus the index rules
// bound to it. The oneof keeps catalog-specific types well-typed
// without forcing the agent to switch on `kind` strings.
message ObjectSnapshot {
string kind = 1; // schema.Kind.String()
string name = 2;
oneof payload {
banyandb.database.v1.Group group_payload = 10;
banyandb.database.v1.Stream stream_payload = 11;
banyandb.database.v1.Measure measure_payload = 12;
banyandb.database.v1.Trace trace_payload = 13;
banyandb.database.v1.IndexRule index_rule_payload = 14;
banyandb.database.v1.IndexRuleBinding index_rule_binding_payload = 15;
banyandb.database.v1.TopNAggregation topn_payload = 16;
}
// bound_index_rules is the set the proto is hashed against. The
// agent uses these directly โ no re-derivation, no extra RPC.
repeated banyandb.database.v1.IndexRule bound_index_rules = 20;
}
```
#### Required: removed proto surface
The `bool include_schema_state = 2` field on `GroupRegistryServiceInspectRequest` (`api/proto/banyandb/database/v1/rpc.proto:482-493`) is removed.
The `repeated ObjectSchemaState schema_objects` field on `DataInfo` (`api/proto/banyandb/database/v1/rpc.proto:533-534`) is removed.
The `repeated ObjectSchemaState schema_objects` field on `LiaisonInfo` (`api/proto/banyandb/database/v1/rpc.proto:626` area) is removed.
The `message ObjectSchemaState` definition itself (`api/proto/banyandb/database/v1/rpc.proto:537-551`) is removed.
#### Acceptance criteria
`StreamGroupSchemaState` is implemented on every data node and liaison that owns a group cache. The server holds a snapshot lock for the stream's duration; every yielded event shares one `snapshot_version`.
Default gRPC message-size limits are not raised. Streaming with bounded per-message `ObjectSnapshot` is the path; unary fallback is not in scope.
The agent opens one stream per data node in parallel; total wall-clock is bounded by the slowest single node, not the sum.
The agent verifies `object_count` on `SchemaSnapshotTrailer` against the received-object count. A mismatch triggers a single retry, then degrades to UNKNOWN for that node.
Wire-compat is preserved end-to-end: new RPC + new messages + new fields are non-breaking additions, and the removed fields are documented in `CHANGES.md` under breaking changes.
Tests cover: empty group, partial failure, large group (โฅ 5,000 objects verifying default message-size limit is not exceeded), mid-stream watch event (verifies version-pinning surfaces as a torn-stream that the agent rejects), and misrouted response (verifies the `node` echo on the trailer matches the requested endpoint identity).
This is a cohort ship with the first comment above โ the architectural shift only works if this RPC exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244#issuecomment-5199635091
Remember to update grafana setup and later oap horizon setup update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] add cluster schema consistency check in FODC lifecycle endpoint [skywalking-banyandb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1244:
URL: https://github.com/apache/skywalking-banyandb/pull/1244#discussion_r3709148195
##########
fodc/proxy/internal/lifecycle/manager.go:
##########
@@ -305,9 +317,117 @@ func (m *Manager) mergeGroups(allData []*agentLifecycleData) []*fodcv1.GroupLife
}
groups = append(groups, g)
}
+ // Range over groupMap is nondeterministic; sort so the payload order is
+ // stable across requests (the checker sorts its own slices for the same
+ // reason).
+ sort.Slice(groups, func(i, j int) bool { return groups[i].GetName() < groups[j].GetName() })
return groups
}
+// mergeSchemaConsistency folds one liaison's verdict into the accumulated one.
+// The status takes the most severe of the two and the issues are the deduped
+// union, so a finding survives no matter which liaison reported it or in what
+// order the agents answered.
+//
+// The result is always a fresh message: the inputs belong to the agents' cached
+// lifecycle data, and mutating them would let one collection round leak into the
+// next.
+func mergeSchemaConsistency(acc, incoming *fodcv1.SchemaConsistency) *fodcv1.SchemaConsistency {
+ if acc == nil && incoming == nil {
+ return nil
+ }
+ merged := &fodcv1.SchemaConsistency{Status: fodcv1.ConsistencyStatus_CONSISTENCY_STATUS_UNSPECIFIED}
+ seenIssue := make(map[string]struct{})
+ // Every liaison's InspectAll produces the full object table for the group, so
+ // dedup by kind+name and union each object's node fingerprints by node id;
+ // the accumulation belongs on a fresh message, never on an agent's cache.
+ objects := make(map[string]*fodcv1.ObjectConsistency)
+ for _, src := range []*fodcv1.SchemaConsistency{acc, incoming} {
+ if src == nil {
+ continue
+ }
+ if consistencySeverity(src.GetStatus()) > consistencySeverity(merged.GetStatus()) {
+ merged.Status = src.GetStatus()
+ }
+ for _, obj := range src.GetObjects() {
+ mergeObject(objects, obj)
+ }
+ for _, issue := range src.GetIssues() {
+ key := schemaIssueKey(issue)
+ if _, dup := seenIssue[key]; dup {
+ continue
+ }
+ seenIssue[key] = struct{}{}
+ merged.Issues = append(merged.Issues, issue)
+ }
+ }
+ // acc/incoming order follows goroutine-completion order; sort so the merged
+ // payload is deterministic regardless of which liaison answered first.
+ merged.Objects = sortedObjects(objects)
+ sort.Slice(merged.Issues, func(i, j int) bool {
+ return schemaIssueKey(merged.Issues[i]) < schemaIssueKey(merged.Issues[j])
+ })
+ return merged
+}
+
+// mergeObject folds one object's record into the accumulator on a fresh
+// ObjectConsistency, unioning node fingerprints by node id.
+func mergeObject(dst map[string]*fodcv1.ObjectConsistency, obj *fodcv1.ObjectConsistency) {
+ key := obj.GetKind() + "|" + obj.GetName()
+ oc, ok := dst[key]
+ if !ok {
+ oc = &fodcv1.ObjectConsistency{
+ Kind: obj.GetKind(), Name: obj.GetName(), RegistryFingerprint: obj.GetRegistryFingerprint(),
+ }
+ dst[key] = oc
+ }
+ seen := make(map[string]struct{}, len(oc.NodeFingerprints))
+ for _, nf := range oc.NodeFingerprints {
+ seen[nf.GetNode()] = struct{}{}
+ }
+ for _, nf := range obj.GetNodeFingerprints() {
+ if _, dup := seen[nf.GetNode()]; dup {
+ continue
+ }
+ seen[nf.GetNode()] = struct{}{}
+ oc.NodeFingerprints = append(oc.NodeFingerprints, nf)
+ }
+}
Review Comment:
mergeObject keeps the registry_fingerprint from whichever liaison/object record is merged first. During rolling upgrades or partial payloads this can incorrectly leave RegistryFingerprint at 0 even when another agent reported a non-zero registry truth for the same (kind,name), reducing the usefulness/correctness of the merged /cluster/lifecycle output. Prefer a non-zero registry fingerprint when available (and ignore nil inputs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org