You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2026/06/06 03:40:07 UTC
[PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]
hanahmily opened a new pull request, #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161
## What
Redesign the `queue_pub` / `queue_sub` Prometheus metrics into a single uniform model that answers the two questions operators actually ask — **which group/operation is slow or erroring**, and **what is the cluster call topology (liaison ↔ data hot/warm/cold)** — instead of the previous ~20 overlapping, `topic`-labeled instruments.
### Metric model
- **Base metrics only:** `total_started`, `total_finished`, `total_latency` (now a **histogram**), `total_err`. Plus **file-sync-only** `sent_bytes` (pub) / `received_bytes` (sub).
- **Labels:** `operation` (`batch-write` / `file-sync` / `query` / `control`), `group`, and remote-endpoint labels `remote_node` / `remote_role` / `remote_tier`; `total_err` adds `error_type`. The `topic` label is removed.
- **Topology:** `remote_node` equals the BanyanDB node `metadata.name`, so a series joins 1:1 with `/cluster/topology` `nodes[]` and `calls[].source/target`. The local end is the scrape target. Each edge is reconstructable from both ends (pub on the source carries `remote_node=<target>`; sub on the target carries `remote_node=<sender>`).
### Wire changes (additive, backward-compatible)
- `cluster.v1.SendRequest` gains `group` (per message) and `sender_node` / `sender_role` / `sender_tier` (stamped on the first frame of a stream).
- `cluster.v1.SyncMetadata` gains `sender_*`.
- Pub-side `remote_role` / `remote_tier` are resolved from the connection registry; sub-side from the wire `sender_*`. Self identity is wired from the metadata current node.
## Breaking change
The previous `queue_*` metric/label names are removed: `*_total_msg_*`, `queue_pub_send_*`, the inflight/retry/backoff gauges, `chunked_sync_*` / `chunk_reorder_*`, and the `topic` label. Dashboards/alerts referencing them must be updated. (CHANGES.md updated.)
## Verification
- `make pre-push` clean for this change: `generate` + `generate-test-cases` + `check` leave the tree consistent (no codegen drift), and `lint` passes. (`vuln-check` flags a pre-existing repo-wide advisory in the indirect dep `golang.org/x/crypto@v0.49.0`, unrelated to this PR.)
- Unit tests: `go test ./banyand/queue/... ./api/data/...` green.
- **Live cluster (SkyWalking showcase):** deployed and confirmed end-to-end — new families present with the new labels, all old families gone, all four `operation` values flowing, `remote_node` matches `/cluster/topology` exactly, and the liaison↔data(hot) edge reconstructs from both the pub (`remote_role=data`, `remote_tier=hot`) and sub (`remote_role=liaison`) sides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161#discussion_r3366676566
##########
banyand/queue/sub/sub.go:
##########
@@ -78,37 +73,49 @@ func checkVersionCompatibility(versionInfo *clusterv1.VersionInfo) (*clusterv1.V
switch {
case !apiCompatible && !fileFormatCompatible:
- versionCompatibility.Supported = false
- versionCompatibility.Reason = fmt.Sprintf("API version %s not supported (server: %s) and file format version %s not compatible (server: %s, supported: %v)",
+ vc.Supported = false
+ vc.Reason = fmt.Sprintf("API version %s not supported (server: %s) and file format version %s not compatible (server: %s, supported: %v)",
versionInfo.ApiVersion, serverAPIVersion, versionInfo.FileFormatVersion, serverFileFormatVersion, compatibleFileFormatVersions)
- return versionCompatibility, modelv1.Status_STATUS_VERSION_UNSUPPORTED
+ return vc, modelv1.Status_STATUS_VERSION_UNSUPPORTED
case !apiCompatible:
- versionCompatibility.Supported = false
- versionCompatibility.Reason = fmt.Sprintf("API version %s not supported (server: %s)", versionInfo.ApiVersion, serverAPIVersion)
- return versionCompatibility, modelv1.Status_STATUS_VERSION_UNSUPPORTED
+ vc.Supported = false
+ vc.Reason = fmt.Sprintf("API version %s not supported (server: %s)", versionInfo.ApiVersion, serverAPIVersion)
+ return vc, modelv1.Status_STATUS_VERSION_UNSUPPORTED
case !fileFormatCompatible:
- versionCompatibility.Supported = false
- versionCompatibility.Reason = fmt.Sprintf("File format version %s not compatible (server: %s, supported: %v)",
+ vc.Supported = false
+ vc.Reason = fmt.Sprintf("File format version %s not compatible (server: %s, supported: %v)",
versionInfo.FileFormatVersion, serverFileFormatVersion, compatibleFileFormatVersions)
- return versionCompatibility, modelv1.Status_STATUS_VERSION_UNSUPPORTED
+ return vc, modelv1.Status_STATUS_VERSION_UNSUPPORTED
}
- versionCompatibility.Supported = true
- versionCompatibility.Reason = "Client version compatible with server"
- return versionCompatibility, modelv1.Status_STATUS_SUCCEED
+ vc.Supported = true
+ vc.Reason = "Client version compatible with server"
+ return vc, modelv1.Status_STATUS_SUCCEED
+}
+
+// streamIdentity captures the per-stream remote sender identity, pinned on the first message.
+type streamIdentity struct {
+ senderNode string
+ senderRole string
+ senderTier string
+ group string
+ operation string
+ pinned bool
}
func (s *server) Send(stream clusterv1.Service_SendServer) error {
ctx := stream.Context()
var topic *bus.Topic
- var m bus.Message
var dataCollection []any
start := time.Now()
+ identity := &streamIdentity{}
+
defer func() {
- if topic != nil {
- s.metrics.totalFinished.Inc(1, topic.String())
- s.metrics.totalLatency.Inc(time.Since(start).Seconds(), topic.String())
+ if topic == nil || !identity.pinned {
+ return
}
+ s.metrics.totalFinished.Inc(1, identity.operation, identity.group, identity.senderNode, identity.senderRole, identity.senderTier)
+ s.metrics.totalLatency.Observe(time.Since(start).Seconds(), identity.operation, identity.group, identity.senderNode, identity.senderRole, identity.senderTier)
Review Comment:
The Send() defer records metrics via s.metrics without checking whether metrics are initialized. Tests (and potentially other call sites) can construct a server without running PreRun(), leaving s.metrics nil, which would panic here.
##########
banyand/queue/sub/helpers.go:
##########
@@ -29,28 +29,28 @@ import (
"github.com/apache/skywalking-banyandb/pkg/logger"
)
-func (s *server) handleEOF(stream clusterv1.Service_SendServer, topic *bus.Topic, dataCollection []any, writeEntity *clusterv1.SendRequest) {
+func (s *server) handleEOF(stream clusterv1.Service_SendServer, topic *bus.Topic, dataCollection []any, writeEntity *clusterv1.SendRequest, identity *streamIdentity) {
if len(dataCollection) < 1 {
return
}
listeners := s.getListeners(*topic)
if len(listeners) == 0 {
- s.reply(stream, writeEntity, nil, "no listener found")
+ s.replyWithErrType(stream, writeEntity, nil, "no listener found", identity, "no_listener")
return
}
if len(listeners) > 1 {
logger.Panicf("multiple listeners found for topic %s", *topic)
}
listener := listeners[0]
if le := listener.CheckHealth(); le != nil {
- s.reply(stream, writeEntity, le, "")
+ s.replyWithErrType(stream, writeEntity, le, "", identity, "handler_error")
return
}
message := listener.Rev(stream.Context(), bus.NewMessage(bus.MessageID(0), dataCollection))
var resp *clusterv1.SendResponse
- data := message.Data()
- if data != nil {
- switch d := data.(type) {
+ msgData := message.Data()
+ if msgData != nil {
+ switch d := msgData.(type) {
case *common.Error:
resp = &clusterv1.SendResponse{
MessageId: writeEntity.MessageId,
Review Comment:
handleEOF can send a nil *SendResponse when listener.Rev returns a message with nil data (resp remains nil), and it also dereferences writeEntity.MessageId even though writeEntity can be nil when Recv() returns io.EOF. Both can lead to panics during batch flush on stream close.
##########
banyand/queue/pub/batch.go:
##########
@@ -50,13 +50,12 @@ const (
// Remote-side failures (observed after the frame was written to the stream).
sendErrReasonRecvError = "recv_error" // s.Recv returned an error (connection/protocol layer).
sendErrReasonServerRejected = "server_rejected" // Server responded with a non-empty Error (includes failover statuses).
-
- sendResultSuccess = "success"
)
type writeStream struct {
- client clusterv1.Service_SendClient
- ctxDoneCh <-chan struct{}
+ client clusterv1.Service_SendClient
+ ctxDoneCh <-chan struct{}
+ isFirstSent bool // true until the first successful send on this stream
}
Review Comment:
The isFirstSent field comment is backwards: the zero value is false, and the code treats false as "first frame not yet sent". As written, the comment says the opposite, which is misleading when maintaining the sender_* stamping logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161#issuecomment-4637282850
## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `4 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 41.16%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5f71a88`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/5f71a885e34058cc82e4ad4c01a0550e26494cf4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 279 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [pkg/cmdsetup/liaison.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?src=pr&el=tree&filepath=pkg%2Fcmdsetup%2Fliaison.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL2NtZHNldHVwL2xpYWlzb24uZ28=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #1161 +/- ##
==========================================
- Coverage 45.97% 41.16% -4.82%
==========================================
Files 328 196 -132
Lines 55505 20384 -35121
==========================================
- Hits 25520 8391 -17129
+ Misses 27909 11143 -16766
+ Partials 2076 850 -1226
```
| [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.16% <0.00%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161#issuecomment-4637469649
Thanks for the review — addressed all three in `e0cb1bb8`:
1. **`sub.go` Send defer / `replyWithErrType` nil metrics** — both now nil-guard `s.metrics`, matching the other recording sites (a server constructed without `PreRun()` has nil metrics).
2. **`helpers.go` `handleEOF` nil response / nil `writeEntity`** — `writeEntity` is nil when the stream closes (`Recv` → `io.EOF`), so the batch flush now reads `MessageId` safely and always sends a non-nil `SendResponse`. (`replyWithErrType` got the same guard since `handleEOF` calls it.)
3. **`batch.go` backwards comment** — renamed `isFirstSent` → `firstFrameSent` with a corrected comment (false until the first frame, which carries `sender_*`, is sent).
Build, lint, and `go test ./banyand/queue/... ./api/data/...` are green; the batch-flush path is exercised by the `banyand/queue/test` integration suite. Also merged latest `main` (incl. #1160).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]
Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org