[PR] Optimize trace pipeline merge performance [skywalking-banyandb]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily opened a new pull request, #1253:
URL: https://github.com/apache/skywalking-banyandb/pull/1253

   ## What changed
   
   - add a resource-limited, production-shaped trace merge benchmark with frozen controlled seeds, logical ledgers, profiling, and HTML reporting
   - account for the complete trace-sampling staging footprint and enforce independent byte, per-trace, and trace-count limits
   - stage raw trace payloads in contiguous arenas and use contiguous metadata descriptors
   - build logical trace groups incrementally so all physical blocks for a trace are evaluated once
   - reuse bounded staging, grouping, evaluation, guard-range, and decision vectors
   - preserve fail-open behavior and avoid recycling decision storage while a timed-out plugin may still access it
   - document the performance test and phased optimization plan
   
   ## Why
   
   The retain-all trace sampling path added substantial allocation and peak-memory overhead during mature merges. Existing measurements were not sufficiently controlled to attribute that cost or safely tune it. This change establishes a reproducible baseline and removes the dominant per-value and per-tag staging allocations without weakening trace completeness, ordering, fragment-guard, or timeout guarantees.
   
   ## Impact
   
   The focused raw staging benchmark improves from 23,856 B/op and 125 allocs/op to 5,552 B/op and 8 allocs/op. A controlled diagnostic merge preserved all 147,126 rows and all core and secondary-index ledgers while reducing the original retain-all medians by approximately 5.8% allocated bytes, 15.5% allocations, 19.6% peak heap, and 22.6% peak RSS.
   
   The remaining performance target stays open: the default 512 MiB staging budget produces one large decision batch, so the next phase will sweep smaller bounded batch sizes to increase within-run reuse.
   
   ## Validation
   
   - `go test ./pkg/pipeline/sdk ./banyand/trace ./banyand/internal/benchmark/...`
   - `go test -race ./pkg/pipeline/sdk ./banyand/trace -count=1`
   - `golangci-lint run ./pkg/pipeline/sdk ./banyand/trace ./banyand/internal/benchmark/...`
   - `go build ./banyand/cmd/trace-merge-benchmark`
   - `git diff --check HEAD`
   - five-pair resource-limited Docker controlled comparison plus a bounded-pool diagnostic rerun
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Optimize trace pipeline merge performance [skywalking-banyandb]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily merged PR #1253:
URL: https://github.com/apache/skywalking-banyandb/pull/1253


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Optimize trace pipeline merge performance [skywalking-banyandb]

["Copilot (via GitHub)" <gi...@apache.org>]

Copilot commented on code in PR #1253:
URL: https://github.com/apache/skywalking-banyandb/pull/1253#discussion_r3725929288


##########
scripts/run-trace-merge-controlled-seed.sh:
##########
@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more contributor
+# license agreements. See the NOTICE file distributed with this work for
+# additional information regarding copyright ownership. The ASF licenses this
+# file to you under the Apache License, Version 2.0 (the "License"); you may not
+# use this file except in compliance with the License. You may obtain a copy of
+# the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+# Capture a frozen controlled merge seed from the production-shaped fixture.
+# The seed underpins the Phase 1 disabled/enabled alternating comparison: every
+# alternating run clones the same seed, so any difference in resource use is
+# attributable to the pipeline rather than to selection drift.
+set -euo pipefail
+
+ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+FIXTURE=${FIXTURE:-"$ROOT/.scratch/trace-pipeline-merge-performance/generated-fixture-ticket05"}
+OUTPUT=${OUTPUT:-"$ROOT/.scratch/trace-pipeline-merge-performance/baseline-report"}
+IMAGE=${IMAGE:-golang:1.25.12}
+DATA_CPUS=${DATA_CPUS:-0-3}
+SEED_OUT=${SEED_OUT:-"$OUTPUT/controlled-seed"}
+BIN="$ROOT/.scratch/trace-pipeline-merge-performance/bin/trace-merge-benchmark"
+
+if [[ ! -x "$BIN" && ! -f "$BIN" ]]; then
+  mkdir -p "$(dirname "$BIN")"
+  go build -o "$BIN" "$ROOT/banyand/cmd/trace-merge-benchmark"
+fi

Review Comment:
   The binary rebuild guard uses `[[ ! -x "$BIN" && ! -f "$BIN" ]]`, which only rebuilds when the path is both non-executable *and* missing. If the file exists but is non-executable/corrupted, the script will skip rebuilding and then fail when the container tries to execute it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org