You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "tristaZero (via GitHub)" <gi...@apache.org> on 2026/09/23 03:44:26 UTC
[I] [BanyanDB] A node that connects to a schema server later never receives the older schema on it [skywalking]
tristaZero opened a new issue, #14103:
URL: https://github.com/apache/skywalking/issues/14103
### Summary
A node keeps one revision watermark for all schema servers. When it starts a watch session with a schema server it finds later, it asks only for schema newer than that watermark. Schema that is only on the new server, and older than the watermark, is skipped. The node gets it at the next full reconcile, which is 150 s later with the default settings.
Until then the node does not know the schema. A distributed query that reaches the node fails with `stream doesn't exist`, and the liaison returns an error for the whole query.
We found this with the integration tests for native self-stored logs (apache/skywalking-banyandb#1368), which are the first tests where a data node creates schema by itself.
### Environment
- BanyanDB cluster: three data nodes and one liaison, every node started as a separate process. Each data node runs a schema server (`schema-registry-mode=property`, the default).
- The files involved (`banyand/metadata/schema/property/client.go`, `converter.go`, `banyand/dquery/stream.go`) are identical to apache/skywalking-banyandb `main` at [`22b2e7b`](https://github.com/apache/skywalking-banyandb/commit/22b2e7b019f8a05cc1fad2896f9052db2f20fc72). The branch of #1368 does not change them.
- Default `--schema-property-client-sync-interval` (30 s) and `DefaultFullReconcileEvery` (5).
- Reproduced on macOS. The code path does not depend on the platform.
### How to reproduce
1. Start data node A and data node B with `--logging-native-enabled`. Each one creates the group `_monitoring_log` and the stream `log` on its own schema server, because it is the only schema server it knows at that moment.
2. Start data node C without that flag. C creates its own schema, for example `_property_gossip`, with a newer revision than the `_monitoring_log` schema on A and B.
3. Start a liaison.
4. Query the stream `log` in group `_monitoring_log` through the liaison.
The query fails for 150 s. C answers `stream doesn't exist`.
### Evidence
From the logs of the run (node C is `127.0.0.1:59967`, started at 19:39:27):
| Time | Log | Node |
|---|---|---|
| 19:39:25 | A creates `_monitoring_log`, revision `1790077166712395000` | A |
| 19:39:27 | `processWatchSession: sending initial replay request`, `maxRevision: 1790077167719535000` | C, watching A |
| 19:39:30 → 19:41:56 | 146 × `fail to get execution context for stream log: stream doesn't exist` | C |
| 19:41:57 | `performFullSync: fetching updated schema from server`, then `creating a tsdb` for `_monitoring_log` | C |
C's watermark (`...167719535000`) is newer than the revision of the schema on A (`...166712395000`), so the replay skipped it. 150 s is five sync rounds of 30 s.
### Cause
References are to `main` at `22b2e7b`.
- A new watch session sends the client's global watermark: `maxRev := r.cache.GetMaxRevision()` ([`client.go#L1436-L1440`](https://github.com/apache/skywalking-banyandb/blob/22b2e7b019f8a05cc1fad2896f9052db2f20fc72/banyand/metadata/schema/property/client.go#L1436-L1440)). The watermark is one value for all servers ([`cache.go#L165`](https://github.com/apache/skywalking-banyandb/blob/22b2e7b019f8a05cc1fad2896f9052db2f20fc72/banyand/metadata/schema/property/cache.go#L165)).
- The criteria is `updated_at > sinceRevision` ([`converter.go#L228`](https://github.com/apache/skywalking-banyandb/blob/22b2e7b019f8a05cc1fad2896f9052db2f20fc72/banyand/metadata/schema/property/converter.go#L228)), so the server sends only newer schema. A resource that is only on this server, and older, is never sent.
- Only the full reconcile repairs it. It runs every fifth sync round ([`client.go#L1306-L1308`](https://github.com/apache/skywalking-banyandb/blob/22b2e7b019f8a05cc1fad2896f9052db2f20fc72/banyand/metadata/schema/property/client.go#L1306-L1308)), that is `DefaultFullReconcileEvery` (5) × `DefaultSyncInterval` (30 s) = 150 s.
- A distributed stream query fails if one data node cannot build the execution context ([`dquery/stream.go#L64`](https://github.com/apache/skywalking-banyandb/blob/22b2e7b019f8a05cc1fad2896f9052db2f20fc72/banyand/dquery/stream.go#L64)), so one node without the schema fails the query for every client.
The same shape applies to any schema that exists on a subset of schema servers, not only to native logs. Schema created through the liaison is written to all of them, so it is not affected.
### Impact
- A cluster where some data nodes have native logging off cannot query `_monitoring_log` for up to 150 s after start.
- A node that joins later, or that reconnects to a server after the server was unreachable, has the same gap for any schema that is only on that server.
- The integration tests for #1368 must set `--schema-property-client-sync-interval=1s` and wait until the stream is queryable before the cases run. Without that, the cases fail for 150 s.
### Proposed solution
In a separate PR:
1. Keep the watermark per schema server, not one for all servers. A new session then replays everything the client has not seen from that server.
2. Or, simpler, send no revision criteria for the first session with a server the client has not synchronized with yet, so the server replays all of its schema.
3. Add a test: create schema on one schema server only, connect a second node, and check that the node has the schema without waiting for a full reconcile.
Alternatives we considered:
- **Make the full reconcile more frequent.** This lowers the delay but does not remove it, and it adds load for every node.
- **Make the data node create the schema on all schema servers.** This does not help a node that joins later, and it does not help any other schema.
### Questions
1. Is the per-server watermark the correct fix, or should a new session always replay in full?
2. Should a distributed query fail when one data node does not know the schema, or should the liaison treat that node as having no data for the query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BanyanDB] A node that connects to a schema server later never receives the older schema on it [skywalking]
Posted by "tristaZero (via GitHub)" <gi...@apache.org>.
tristaZero commented on issue #14103:
URL: https://github.com/apache/skywalking/issues/14103#issuecomment-5792005422
Update on the test impact: the cluster suite of apache/skywalking-banyandb#1368 now runs one data node in-process, so it no longer needs the `--schema-property-client-sync-interval=1s` workaround, and that line is gone. The defect itself is unchanged: a node that connects to a schema server later replays only revisions newer than its own watermark, so it misses older schema that exists only on that server until a full reconcile, 150 s with the defaults.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BanyanDB] A node that connects to a schema server later never receives the older schema on it [skywalking]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily closed issue #14103: [BanyanDB] A node that connects to a schema server later never receives the older schema on it
URL: https://github.com/apache/skywalking/issues/14103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BanyanDB] A node that connects to a schema server later never receives the older schema on it [skywalking]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #14103:
URL: https://github.com/apache/skywalking/issues/14103#issuecomment-5789221326
Closing this schema-sync proposal in favor of #14104, which addresses the query-path behavior.
Schema convergence through incremental updates and periodic full reconciliation is intended behavior. We do not need to change the revision watermark or replay policy for this case. Instead, when the liaison knows the requested schema but a participating data node does not yet know it, distributed stream queries should handle that specific absence gracefully and return the available results from other nodes. Unrelated query failures and liaison-local schema-not-found errors must retain their existing handling.
The replacement issue defines the query semantics, convergence trade-off, and focused regression coverage. This closure does not mean that the query-path improvement has already been implemented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org