You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2026/06/06 03:40:07 UTC

[PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]

hanahmily opened a new pull request, #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161

   ## What
   
   Redesign the `queue_pub` / `queue_sub` Prometheus metrics into a single uniform model that answers the two questions operators actually ask — **which group/operation is slow or erroring**, and **what is the cluster call topology (liaison ↔ data hot/warm/cold)** — instead of the previous ~20 overlapping, `topic`-labeled instruments.
   
   ### Metric model
   - **Base metrics only:** `total_started`, `total_finished`, `total_latency` (now a **histogram**), `total_err`. Plus **file-sync-only** `sent_bytes` (pub) / `received_bytes` (sub).
   - **Labels:** `operation` (`batch-write` / `file-sync` / `query` / `control`), `group`, and remote-endpoint labels `remote_node` / `remote_role` / `remote_tier`; `total_err` adds `error_type`. The `topic` label is removed.
   - **Topology:** `remote_node` equals the BanyanDB node `metadata.name`, so a series joins 1:1 with `/cluster/topology` `nodes[]` and `calls[].source/target`. The local end is the scrape target. Each edge is reconstructable from both ends (pub on the source carries `remote_node=<target>`; sub on the target carries `remote_node=<sender>`).
   
   ### Wire changes (additive, backward-compatible)
   - `cluster.v1.SendRequest` gains `group` (per message) and `sender_node` / `sender_role` / `sender_tier` (stamped on the first frame of a stream).
   - `cluster.v1.SyncMetadata` gains `sender_*`.
   - Pub-side `remote_role` / `remote_tier` are resolved from the connection registry; sub-side from the wire `sender_*`. Self identity is wired from the metadata current node.
   
   ## Breaking change
   The previous `queue_*` metric/label names are removed: `*_total_msg_*`, `queue_pub_send_*`, the inflight/retry/backoff gauges, `chunked_sync_*` / `chunk_reorder_*`, and the `topic` label. Dashboards/alerts referencing them must be updated. (CHANGES.md updated.)
   
   ## Verification
   - `make pre-push` clean for this change: `generate` + `generate-test-cases` + `check` leave the tree consistent (no codegen drift), and `lint` passes. (`vuln-check` flags a pre-existing repo-wide advisory in the indirect dep `golang.org/x/crypto@v0.49.0`, unrelated to this PR.)
   - Unit tests: `go test ./banyand/queue/... ./api/data/...` green.
   - **Live cluster (SkyWalking showcase):** deployed and confirmed end-to-end — new families present with the new labels, all old families gone, all four `operation` values flowing, `remote_node` matches `/cluster/topology` exactly, and the liaison↔data(hot) edge reconstructs from both the pub (`remote_role=data`, `remote_tier=hot`) and sub (`remote_role=liaison`) sides.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161#discussion_r3366676566


##########
banyand/queue/sub/sub.go:
##########
@@ -78,37 +73,49 @@ func checkVersionCompatibility(versionInfo *clusterv1.VersionInfo) (*clusterv1.V
 
 	switch {
 	case !apiCompatible && !fileFormatCompatible:
-		versionCompatibility.Supported = false
-		versionCompatibility.Reason = fmt.Sprintf("API version %s not supported (server: %s) and file format version %s not compatible (server: %s, supported: %v)",
+		vc.Supported = false
+		vc.Reason = fmt.Sprintf("API version %s not supported (server: %s) and file format version %s not compatible (server: %s, supported: %v)",
 			versionInfo.ApiVersion, serverAPIVersion, versionInfo.FileFormatVersion, serverFileFormatVersion, compatibleFileFormatVersions)
-		return versionCompatibility, modelv1.Status_STATUS_VERSION_UNSUPPORTED
+		return vc, modelv1.Status_STATUS_VERSION_UNSUPPORTED
 	case !apiCompatible:
-		versionCompatibility.Supported = false
-		versionCompatibility.Reason = fmt.Sprintf("API version %s not supported (server: %s)", versionInfo.ApiVersion, serverAPIVersion)
-		return versionCompatibility, modelv1.Status_STATUS_VERSION_UNSUPPORTED
+		vc.Supported = false
+		vc.Reason = fmt.Sprintf("API version %s not supported (server: %s)", versionInfo.ApiVersion, serverAPIVersion)
+		return vc, modelv1.Status_STATUS_VERSION_UNSUPPORTED
 	case !fileFormatCompatible:
-		versionCompatibility.Supported = false
-		versionCompatibility.Reason = fmt.Sprintf("File format version %s not compatible (server: %s, supported: %v)",
+		vc.Supported = false
+		vc.Reason = fmt.Sprintf("File format version %s not compatible (server: %s, supported: %v)",
 			versionInfo.FileFormatVersion, serverFileFormatVersion, compatibleFileFormatVersions)
-		return versionCompatibility, modelv1.Status_STATUS_VERSION_UNSUPPORTED
+		return vc, modelv1.Status_STATUS_VERSION_UNSUPPORTED
 	}
 
-	versionCompatibility.Supported = true
-	versionCompatibility.Reason = "Client version compatible with server"
-	return versionCompatibility, modelv1.Status_STATUS_SUCCEED
+	vc.Supported = true
+	vc.Reason = "Client version compatible with server"
+	return vc, modelv1.Status_STATUS_SUCCEED
+}
+
+// streamIdentity captures the per-stream remote sender identity, pinned on the first message.
+type streamIdentity struct {
+	senderNode string
+	senderRole string
+	senderTier string
+	group      string
+	operation  string
+	pinned     bool
 }
 
 func (s *server) Send(stream clusterv1.Service_SendServer) error {
 	ctx := stream.Context()
 	var topic *bus.Topic
-	var m bus.Message
 	var dataCollection []any
 	start := time.Now()
+	identity := &streamIdentity{}
+
 	defer func() {
-		if topic != nil {
-			s.metrics.totalFinished.Inc(1, topic.String())
-			s.metrics.totalLatency.Inc(time.Since(start).Seconds(), topic.String())
+		if topic == nil || !identity.pinned {
+			return
 		}
+		s.metrics.totalFinished.Inc(1, identity.operation, identity.group, identity.senderNode, identity.senderRole, identity.senderTier)
+		s.metrics.totalLatency.Observe(time.Since(start).Seconds(), identity.operation, identity.group, identity.senderNode, identity.senderRole, identity.senderTier)

Review Comment:
   The Send() defer records metrics via s.metrics without checking whether metrics are initialized. Tests (and potentially other call sites) can construct a server without running PreRun(), leaving s.metrics nil, which would panic here.



##########
banyand/queue/sub/helpers.go:
##########
@@ -29,28 +29,28 @@ import (
 	"github.com/apache/skywalking-banyandb/pkg/logger"
 )
 
-func (s *server) handleEOF(stream clusterv1.Service_SendServer, topic *bus.Topic, dataCollection []any, writeEntity *clusterv1.SendRequest) {
+func (s *server) handleEOF(stream clusterv1.Service_SendServer, topic *bus.Topic, dataCollection []any, writeEntity *clusterv1.SendRequest, identity *streamIdentity) {
 	if len(dataCollection) < 1 {
 		return
 	}
 	listeners := s.getListeners(*topic)
 	if len(listeners) == 0 {
-		s.reply(stream, writeEntity, nil, "no listener found")
+		s.replyWithErrType(stream, writeEntity, nil, "no listener found", identity, "no_listener")
 		return
 	}
 	if len(listeners) > 1 {
 		logger.Panicf("multiple listeners found for topic %s", *topic)
 	}
 	listener := listeners[0]
 	if le := listener.CheckHealth(); le != nil {
-		s.reply(stream, writeEntity, le, "")
+		s.replyWithErrType(stream, writeEntity, le, "", identity, "handler_error")
 		return
 	}
 	message := listener.Rev(stream.Context(), bus.NewMessage(bus.MessageID(0), dataCollection))
 	var resp *clusterv1.SendResponse
-	data := message.Data()
-	if data != nil {
-		switch d := data.(type) {
+	msgData := message.Data()
+	if msgData != nil {
+		switch d := msgData.(type) {
 		case *common.Error:
 			resp = &clusterv1.SendResponse{
 				MessageId: writeEntity.MessageId,

Review Comment:
   handleEOF can send a nil *SendResponse when listener.Rev returns a message with nil data (resp remains nil), and it also dereferences writeEntity.MessageId even though writeEntity can be nil when Recv() returns io.EOF. Both can lead to panics during batch flush on stream close.



##########
banyand/queue/pub/batch.go:
##########
@@ -50,13 +50,12 @@ const (
 	// Remote-side failures (observed after the frame was written to the stream).
 	sendErrReasonRecvError      = "recv_error"      // s.Recv returned an error (connection/protocol layer).
 	sendErrReasonServerRejected = "server_rejected" // Server responded with a non-empty Error (includes failover statuses).
-
-	sendResultSuccess = "success"
 )
 
 type writeStream struct {
-	client    clusterv1.Service_SendClient
-	ctxDoneCh <-chan struct{}
+	client      clusterv1.Service_SendClient
+	ctxDoneCh   <-chan struct{}
+	isFirstSent bool // true until the first successful send on this stream
 }

Review Comment:
   The isFirstSent field comment is backwards: the zero value is false, and the code treats false as "first frame not yet sent". As written, the comment says the opposite, which is misleading when maintaining the sender_* stamping logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161#issuecomment-4637282850

   ## [Codecov](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `4 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 41.16%. Comparing base ([`3530dd9`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/3530dd9b5cefaa4fef60eec91ecbfbdfcaf243c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5f71a88`](https://app.codecov.io/gh/apache/skywalking-banyandb/commit/5f71a885e34058cc82e4ad4c01a0550e26494cf4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 279 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [pkg/cmdsetup/liaison.go](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?src=pr&el=tree&filepath=pkg%2Fcmdsetup%2Fliaison.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGtnL2NtZHNldHVwL2xpYWlzb24uZ28=) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #1161      +/-   ##
   ==========================================
   - Coverage   45.97%   41.16%   -4.82%     
   ==========================================
     Files         328      196     -132     
     Lines       55505    20384   -35121     
   ==========================================
   - Hits        25520     8391   -17129     
   + Misses      27909    11143   -16766     
   + Partials     2076      850    -1226     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [pkg](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `41.16% <0.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/skywalking-banyandb/pull/1161?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161#issuecomment-4637469649

   Thanks for the review — addressed all three in `e0cb1bb8`:
   
   1. **`sub.go` Send defer / `replyWithErrType` nil metrics** — both now nil-guard `s.metrics`, matching the other recording sites (a server constructed without `PreRun()` has nil metrics).
   2. **`helpers.go` `handleEOF` nil response / nil `writeEntity`** — `writeEntity` is nil when the stream closes (`Recv` → `io.EOF`), so the batch flush now reads `MessageId` safely and always sends a non-nil `SendResponse`. (`replyWithErrType` got the same guard since `handleEOF` calls it.)
   3. **`batch.go` backwards comment** — renamed `isFirstSent` → `firstFrameSent` with a corrected comment (false until the first frame, which carries `sender_*`, is sent).
   
   Build, lint, and `go test ./banyand/queue/... ./api/data/...` are green; the batch-flush path is exercised by the `banyand/queue/test` integration suite. Also merged latest `main` (incl. #1160).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(queue): redesign pub/sub metrics around operation, group, and node topology [skywalking-banyandb]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily merged PR #1161:
URL: https://github.com/apache/skywalking-banyandb/pull/1161


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org