You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by ha...@apache.org on 2026/05/14 06:12:19 UTC
(skywalking-banyandb) 09/50: feat(query/vectorized/measure): add bench gates infrastructure (gate G5a)
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch vectorized-query
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
commit 2614f0981ddab52a97e61499365ccdb7fb3f1374
Author: Hongtao Gao <ha...@gmail.com>
AuthorDate: Wed May 6 15:11:55 2026 +0000
feat(query/vectorized/measure): add bench gates infrastructure (gate G5a)
Implements G5a (M9 — Performance bench gates), the prerequisite for the
default-flip rollout. Microbenchmarks, in-Go gate enforcement, runner
script with markdown report writer, and a macro suite scaffold are in
place; G5b (soak) and G5c (default flip) remain gated on humans.
pkg/query/vectorized/measure:
- bench_test.go: paired BenchmarkRowPath_W{1..5} / BenchmarkVectorizedPath_W{1..5}.
Both paths consume the same fake MeasureQueryResult; tag and field
protobuf values are shared singletons across rows so allocation cost
measures the serializer, not fixture construction. Workload scales
bounded for unit-bench tractability — gates are ratios so the
comparison holds at any scale.
- bench_gates_test.go: TestBenchGates_PerWorkload runs both paths via
testing.Benchmark and asserts the spec's vec/row ratios. RUN_BENCH_GATES=1
to opt in (kept off the default test path so go test ./... stays fast).
scripts/bench-vectorized.sh: drives -bench=. -benchmem -count=N
-benchtime=Xs across W1..W5, runs the gate test, and emits
dist/bench/vectorized-YYYYMMDD-HHMMSS.md with paired output and PASS/FAIL
verdict. Exit code mirrors the gate-test result.
test/integration/standalone/benchmark/: macro-suite scaffold
(suite_test, benchmark_test, workloads_test). The Describe block is
default-skipped behind RUN_VECTORIZED_BENCH=1 because the in-process
end-to-end W3/W4 measurement is meaningful only after the iterator
wires BatchLimit / BatchGroupBy / BatchAggregation / BatchTop into the
live Pipeline (deferred from G4). The microbench gate is the
authoritative G5a check.
.gitignore: ignore dist/bench/ artifacts.
Verification:
bash scripts/bench-vectorized.sh
Currently produces a non-zero exit on this branch -- the vectorized
adapter runs ~1.4-2.2x slower than the row path across W1..W5 and
allocates 1.6-2.6x more, so all five gates fail. That is the spec's
intended signal: G5b/G5c are blocked until perf work lands.
---
.gitignore | 3 +
pkg/query/vectorized/measure/bench_gates_test.go | 115 ++++++++
pkg/query/vectorized/measure/bench_test.go | 309 +++++++++++++++++++++
scripts/bench-vectorized.sh | 117 ++++++++
.../standalone/benchmark/benchmark_suite_test.go | 42 +++
.../standalone/benchmark/benchmark_test.go | 63 +++++
.../standalone/benchmark/workloads_test.go | 60 ++++
7 files changed, 709 insertions(+)
diff --git a/.gitignore b/.gitignore
index 9af8ace51..4785e2c8b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -84,3 +84,6 @@ fodc/agent/internal/ktm/iomonitor/ebpf/generated/vmlinux.h
# OpenSSL serial files
*.srl
+# Vectorized benchmark report artifacts (G5a)
+dist/bench/
+
diff --git a/pkg/query/vectorized/measure/bench_gates_test.go b/pkg/query/vectorized/measure/bench_gates_test.go
new file mode 100644
index 000000000..23182efb7
--- /dev/null
+++ b/pkg/query/vectorized/measure/bench_gates_test.go
@@ -0,0 +1,115 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "os"
+ "testing"
+)
+
+// G5a acceptance gates per spec §"Performance Evaluation Plan". Ratios are
+// vectorized / row; failing the gate is a regression that blocks the
+// default-flip rollout.
+type benchGate struct {
+ id string
+ maxNsRatio float64 // ns/op ≤ row × maxNsRatio
+ maxAllocRatio float64 // allocs ≤ row × maxAllocRatio
+ maxBytesRatio float64 // B/op ≤ row × maxBytesRatio
+}
+
+var benchGates = map[string]benchGate{
+ "W1": {id: "W1", maxNsRatio: 1.05, maxAllocRatio: 1.00, maxBytesRatio: 1.20},
+ "W2": {id: "W2", maxNsRatio: 1.05, maxAllocRatio: 1.00, maxBytesRatio: 1.20},
+ "W3": {id: "W3", maxNsRatio: 1.00, maxAllocRatio: 1.00, maxBytesRatio: 1.20},
+ "W4": {id: "W4", maxNsRatio: 1.05, maxAllocRatio: 1.00, maxBytesRatio: 1.20},
+ "W5": {id: "W5", maxNsRatio: 1.05, maxAllocRatio: 1.00, maxBytesRatio: 1.20},
+}
+
+// TestBenchGates_PerWorkload runs both serialization paths inside testing.B
+// harnesses and asserts the spec's vec/row ratios. A regression fails this
+// test, not just the markdown report — gates are enforced as code.
+//
+// Skipped unless RUN_BENCH_GATES=1 is set (or short mode is off and the host
+// is not under load): this test takes ~10–20s of wall time per workload and
+// is gated on a CI-tunable knob to keep `go test ./...` fast.
+func TestBenchGates_PerWorkload(t *testing.T) {
+ if os.Getenv("RUN_BENCH_GATES") != "1" {
+ t.Skip("set RUN_BENCH_GATES=1 to run G5a bench gates")
+ }
+ if testing.Short() {
+ t.Skip("skipping bench gates in -short mode")
+ }
+ for _, spec := range allWorkloads {
+ t.Run(spec.id, func(t *testing.T) {
+ gate := benchGates[spec.id]
+ row := timeWorkload(spec, false)
+ vec := timeWorkload(spec, true)
+
+ t.Logf("%s row: %d ns/op, %d B/op, %d allocs/op",
+ spec.id, row.NsPerOp(), row.AllocedBytesPerOp(), row.AllocsPerOp())
+ t.Logf("%s vec: %d ns/op, %d B/op, %d allocs/op",
+ spec.id, vec.NsPerOp(), vec.AllocedBytesPerOp(), vec.AllocsPerOp())
+
+ if !ratioLE(vec.NsPerOp(), row.NsPerOp(), gate.maxNsRatio) {
+ t.Fatalf("%s ns/op gate: vec %d > row %d × %.2f (= %d)",
+ spec.id, vec.NsPerOp(), row.NsPerOp(), gate.maxNsRatio,
+ int64(float64(row.NsPerOp())*gate.maxNsRatio))
+ }
+ if !ratioLE(vec.AllocsPerOp(), row.AllocsPerOp(), gate.maxAllocRatio) {
+ t.Fatalf("%s allocs/op gate: vec %d > row %d × %.2f (= %d)",
+ spec.id, vec.AllocsPerOp(), row.AllocsPerOp(), gate.maxAllocRatio,
+ int64(float64(row.AllocsPerOp())*gate.maxAllocRatio))
+ }
+ if !ratioLE(vec.AllocedBytesPerOp(), row.AllocedBytesPerOp(), gate.maxBytesRatio) {
+ t.Fatalf("%s B/op gate: vec %d > row %d × %.2f (= %d)",
+ spec.id, vec.AllocedBytesPerOp(), row.AllocedBytesPerOp(), gate.maxBytesRatio,
+ int64(float64(row.AllocedBytesPerOp())*gate.maxBytesRatio))
+ }
+ })
+ }
+}
+
+// ratioLE reports whether got ≤ baseline × ratio. baseline=0 means we cannot
+// form a meaningful ratio; treat as pass.
+func ratioLE(got, baseline int64, ratio float64) bool {
+ if baseline <= 0 {
+ return true
+ }
+ return float64(got) <= float64(baseline)*ratio
+}
+
+// timeWorkload runs the appropriate path inside a testing.B for ~2 seconds
+// and returns the result. The caller treats vectorized and row identically
+// at the comparison layer.
+func timeWorkload(spec workloadSpec, vectorized bool) testing.BenchmarkResult {
+ results := buildResults(spec)
+ schema := buildSchema(spec)
+ opts := buildOpts(spec)
+ cfg := VectorizedConfig{Enabled: true, BatchSize: 1024, QueryMemoryMiB: 64}
+ return testing.Benchmark(func(b *testing.B) {
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ if vectorized {
+ runVectorizedPath(results, schema, opts, cfg)
+ } else {
+ runRowPath(results, opts)
+ }
+ }
+ })
+}
diff --git a/pkg/query/vectorized/measure/bench_test.go b/pkg/query/vectorized/measure/bench_test.go
new file mode 100644
index 000000000..79563ecbe
--- /dev/null
+++ b/pkg/query/vectorized/measure/bench_test.go
@@ -0,0 +1,309 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "testing"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+)
+
+// Microbenchmarks (G5a) — paired row-path vs vectorized-path benchmarks for
+// W1..W5 per the spec's Performance Evaluation Plan. Both paths consume the
+// same fake MeasureQueryResult; only the serialization implementation
+// differs. ns/op, B/op, allocs/op are reported via testing.B.
+//
+// Workload scales are bounded for unit-bench tractability — the integration
+// macro suite at test/integration/standalone/benchmark/ exercises full-scale
+// shapes against the real Measure module. Acceptance gates are ratios
+// (vec/row), so the relative comparison holds at any scale; absolute
+// throughput is not the gate.
+//
+// Run via:
+//
+// go test ./pkg/query/vectorized/measure -bench=. -benchmem -count=5 -benchtime=2s
+
+// workloadSpec parameterizes a benchmark workload.
+type workloadSpec struct {
+ id string
+ tagFamilies []tagSpec
+ fields []fieldSpec
+ series int
+ rowsPer int
+}
+
+type tagSpec struct {
+ family string
+ name string
+ col databasev1.TagType
+}
+
+type fieldSpec struct {
+ name string
+ col databasev1.FieldType
+}
+
+// w1..w5 mirror the spec's catalog, scaled for unit-bench memory budgets.
+// Total rows per workload is held near 100k so each benchmark iteration
+// completes quickly enough for -benchtime=2s to amortize fixture build cost.
+var (
+ w1 = workloadSpec{
+ id: "W1",
+ series: 1, rowsPer: 10000,
+ fields: []fieldSpec{{name: "v_int", col: databasev1.FieldType_FIELD_TYPE_INT}},
+ }
+ w2 = workloadSpec{
+ id: "W2",
+ series: 100, rowsPer: 1000,
+ tagFamilies: []tagSpec{
+ {family: "default", name: "svc", col: databasev1.TagType_TAG_TYPE_STRING},
+ {family: "default", name: "env_id", col: databasev1.TagType_TAG_TYPE_INT},
+ },
+ fields: []fieldSpec{
+ {name: "v_int", col: databasev1.FieldType_FIELD_TYPE_INT},
+ {name: "v_float", col: databasev1.FieldType_FIELD_TYPE_FLOAT},
+ },
+ }
+ w3 = workloadSpec{
+ id: "W3",
+ series: 1000, rowsPer: 100,
+ tagFamilies: []tagSpec{{family: "default", name: "svc", col: databasev1.TagType_TAG_TYPE_STRING}},
+ fields: []fieldSpec{{name: "v_int", col: databasev1.FieldType_FIELD_TYPE_INT}},
+ }
+ w4 = workloadSpec{
+ id: "W4",
+ series: 100, rowsPer: 1000,
+ tagFamilies: []tagSpec{{family: "default", name: "svc", col: databasev1.TagType_TAG_TYPE_STRING}},
+ fields: []fieldSpec{{name: "v_int", col: databasev1.FieldType_FIELD_TYPE_INT}},
+ }
+ w5 = workloadSpec{
+ id: "W5",
+ series: 1000, rowsPer: 100,
+ tagFamilies: []tagSpec{
+ {family: "default", name: "svc", col: databasev1.TagType_TAG_TYPE_STRING},
+ {family: "default", name: "env_id", col: databasev1.TagType_TAG_TYPE_INT},
+ {family: "default", name: "blob", col: databasev1.TagType_TAG_TYPE_DATA_BINARY},
+ {family: "default", name: "ports", col: databasev1.TagType_TAG_TYPE_INT_ARRAY},
+ {family: "default", name: "labels", col: databasev1.TagType_TAG_TYPE_STRING_ARRAY},
+ },
+ fields: []fieldSpec{
+ {name: "v_int", col: databasev1.FieldType_FIELD_TYPE_INT},
+ {name: "v_float", col: databasev1.FieldType_FIELD_TYPE_FLOAT},
+ {name: "v_str", col: databasev1.FieldType_FIELD_TYPE_STRING},
+ {name: "v_bytes", col: databasev1.FieldType_FIELD_TYPE_DATA_BINARY},
+ },
+ }
+
+ allWorkloads = []workloadSpec{w1, w2, w3, w4, w5}
+)
+
+// buildResults materializes a deterministic []*model.MeasureResult for the
+// workload. Tag and field protobuf values are shared singletons across rows
+// to keep allocation cost concentrated on serialization (the path under test)
+// rather than fixture construction.
+func buildResults(spec workloadSpec) []*model.MeasureResult {
+ tagSingletons := map[string]*modelv1.TagValue{
+ "str": {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "alpha"}}},
+ "int": {Value: &modelv1.TagValue_Int{Int: &modelv1.Int{Value: 7}}},
+ "binary": {Value: &modelv1.TagValue_BinaryData{BinaryData: []byte{0xfe, 0xed}}},
+ "intarr": {Value: &modelv1.TagValue_IntArray{IntArray: &modelv1.IntArray{Value: []int64{1, 2}}}},
+ "strarr": {Value: &modelv1.TagValue_StrArray{StrArray: &modelv1.StrArray{Value: []string{"x", "y"}}}},
+ }
+ fieldSingletons := map[string]*modelv1.FieldValue{
+ "int": {Value: &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: 42}}},
+ "float": {Value: &modelv1.FieldValue_Float{Float: &modelv1.Float{Value: 3.14}}},
+ "str": {Value: &modelv1.FieldValue_Str{Str: &modelv1.Str{Value: "ok"}}},
+ "binary": {Value: &modelv1.FieldValue_BinaryData{BinaryData: []byte{0xab, 0xcd}}},
+ }
+
+ results := make([]*model.MeasureResult, 0, spec.series)
+ for s := 0; s < spec.series; s++ {
+ r := &model.MeasureResult{SID: common.SeriesID(s + 1)}
+ r.Timestamps = make([]int64, spec.rowsPer)
+ r.Versions = make([]int64, spec.rowsPer)
+ r.ShardIDs = make([]common.ShardID, spec.rowsPer)
+ for i := 0; i < spec.rowsPer; i++ {
+ r.Timestamps[i] = int64(i)
+ r.Versions[i] = 1
+ }
+ if len(spec.tagFamilies) > 0 {
+ tags := make([]model.Tag, 0, len(spec.tagFamilies))
+ for _, ts := range spec.tagFamilies {
+ values := make([]*modelv1.TagValue, spec.rowsPer)
+ singleton := pickTagSingleton(ts.col, tagSingletons)
+ for i := range values {
+ values[i] = singleton
+ }
+ tags = append(tags, model.Tag{Name: ts.name, Values: values})
+ }
+ r.TagFamilies = []model.TagFamily{{Name: spec.tagFamilies[0].family, Tags: tags}}
+ }
+ if len(spec.fields) > 0 {
+ r.Fields = make([]model.Field, 0, len(spec.fields))
+ for _, f := range spec.fields {
+ values := make([]*modelv1.FieldValue, spec.rowsPer)
+ singleton := pickFieldSingleton(f.col, fieldSingletons)
+ for i := range values {
+ values[i] = singleton
+ }
+ r.Fields = append(r.Fields, model.Field{Name: f.name, Values: values})
+ }
+ }
+ results = append(results, r)
+ }
+ return results
+}
+
+func pickTagSingleton(t databasev1.TagType, m map[string]*modelv1.TagValue) *modelv1.TagValue {
+ switch t {
+ case databasev1.TagType_TAG_TYPE_INT:
+ return m["int"]
+ case databasev1.TagType_TAG_TYPE_STRING:
+ return m["str"]
+ case databasev1.TagType_TAG_TYPE_DATA_BINARY:
+ return m["binary"]
+ case databasev1.TagType_TAG_TYPE_INT_ARRAY:
+ return m["intarr"]
+ case databasev1.TagType_TAG_TYPE_STRING_ARRAY:
+ return m["strarr"]
+ case databasev1.TagType_TAG_TYPE_UNSPECIFIED, databasev1.TagType_TAG_TYPE_TIMESTAMP:
+ // Bench fixtures never use these variants.
+ return m["str"]
+ }
+ return m["str"]
+}
+
+func pickFieldSingleton(t databasev1.FieldType, m map[string]*modelv1.FieldValue) *modelv1.FieldValue {
+ switch t {
+ case databasev1.FieldType_FIELD_TYPE_INT:
+ return m["int"]
+ case databasev1.FieldType_FIELD_TYPE_FLOAT:
+ return m["float"]
+ case databasev1.FieldType_FIELD_TYPE_STRING:
+ return m["str"]
+ case databasev1.FieldType_FIELD_TYPE_DATA_BINARY:
+ return m["binary"]
+ }
+ return m["int"]
+}
+
+// buildSchema reconstructs the *databasev1.Measure schema corresponding to
+// the workload, only declaring the tags and fields that the workload
+// projects.
+func buildSchema(spec workloadSpec) *databasev1.Measure {
+ m := &databasev1.Measure{}
+ if len(spec.tagFamilies) > 0 {
+ fam := &databasev1.TagFamilySpec{Name: spec.tagFamilies[0].family}
+ for _, t := range spec.tagFamilies {
+ fam.Tags = append(fam.Tags, &databasev1.TagSpec{Name: t.name, Type: t.col})
+ }
+ m.TagFamilies = []*databasev1.TagFamilySpec{fam}
+ }
+ for _, f := range spec.fields {
+ m.Fields = append(m.Fields, &databasev1.FieldSpec{Name: f.name, FieldType: f.col})
+ }
+ return m
+}
+
+// buildOpts derives MeasureQueryOptions matching the workload's projection.
+func buildOpts(spec workloadSpec) model.MeasureQueryOptions {
+ opts := model.MeasureQueryOptions{}
+ if len(spec.tagFamilies) > 0 {
+ names := make([]string, 0, len(spec.tagFamilies))
+ for _, t := range spec.tagFamilies {
+ names = append(names, t.name)
+ }
+ opts.TagProjection = []model.TagProjection{{Family: spec.tagFamilies[0].family, Names: names}}
+ }
+ if len(spec.fields) > 0 {
+ opts.FieldProjection = make([]string, 0, len(spec.fields))
+ for _, f := range spec.fields {
+ opts.FieldProjection = append(opts.FieldProjection, f.name)
+ }
+ }
+ return opts
+}
+
+// benchSink prevents the compiler from eliding the work inside benchmark
+// loops — every drained row is summed into it.
+var benchSink int
+
+// runRowPath drains the row-path serializer once over a fresh cursor backed
+// by the supplied results, accumulating the row count into benchSink.
+func runRowPath(results []*model.MeasureResult, opts model.MeasureQueryOptions) {
+ qr := &fakeMeasureQueryResult{seq: results}
+ benchSink += len(rowSerialize(qr, opts))
+}
+
+// runVectorizedPath drains the vectorized adapter once over a fresh cursor.
+func runVectorizedPath(results []*model.MeasureResult, schema *databasev1.Measure,
+ opts model.MeasureQueryOptions, cfg VectorizedConfig,
+) {
+ qr := &fakeMeasureQueryResult{seq: results}
+ it, err := NewMIterator(context.Background(), qr, schema, opts, cfg)
+ if err != nil {
+ panic(err)
+ }
+ defer it.Close()
+ for it.Next() {
+ benchSink++
+ }
+}
+
+func benchmarkRow(b *testing.B, spec workloadSpec) {
+ results := buildResults(spec)
+ opts := buildOpts(spec)
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ runRowPath(results, opts)
+ }
+}
+
+func benchmarkVectorized(b *testing.B, spec workloadSpec) {
+ results := buildResults(spec)
+ schema := buildSchema(spec)
+ opts := buildOpts(spec)
+ cfg := VectorizedConfig{Enabled: true, BatchSize: 1024, QueryMemoryMiB: 64}
+ b.ReportAllocs()
+ b.ResetTimer()
+ for i := 0; i < b.N; i++ {
+ runVectorizedPath(results, schema, opts, cfg)
+ }
+}
+
+// Paired benchmarks per spec.
+
+func BenchmarkRowPath_W1(b *testing.B) { benchmarkRow(b, w1) }
+func BenchmarkVectorizedPath_W1(b *testing.B) { benchmarkVectorized(b, w1) }
+
+func BenchmarkRowPath_W2(b *testing.B) { benchmarkRow(b, w2) }
+func BenchmarkVectorizedPath_W2(b *testing.B) { benchmarkVectorized(b, w2) }
+
+func BenchmarkRowPath_W3(b *testing.B) { benchmarkRow(b, w3) }
+func BenchmarkVectorizedPath_W3(b *testing.B) { benchmarkVectorized(b, w3) }
+
+func BenchmarkRowPath_W4(b *testing.B) { benchmarkRow(b, w4) }
+func BenchmarkVectorizedPath_W4(b *testing.B) { benchmarkVectorized(b, w4) }
+
+func BenchmarkRowPath_W5(b *testing.B) { benchmarkRow(b, w5) }
+func BenchmarkVectorizedPath_W5(b *testing.B) { benchmarkVectorized(b, w5) }
diff --git a/scripts/bench-vectorized.sh b/scripts/bench-vectorized.sh
new file mode 100755
index 000000000..b51317075
--- /dev/null
+++ b/scripts/bench-vectorized.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# Licensed to Apache Software Foundation (ASF) under one or more contributor
+# license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright
+# ownership. Apache Software Foundation (ASF) licenses this file to you under
+# the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# G5a runner — drives the paired W1..W5 microbenchmarks, runs the in-Go gate
+# enforcement test, and emits a markdown report at
+# dist/bench/vectorized-YYYYMMDD-HHMMSS.md. Exit non-zero if any gate fails.
+#
+# Usage:
+# ./scripts/bench-vectorized.sh
+# COUNT=3 BENCHTIME=1s ./scripts/bench-vectorized.sh # quicker sanity run
+#
+# Macro benchmarks live at test/integration/standalone/benchmark/ and run
+# separately via `make test-ci PKG=./test/integration/standalone/benchmark/...`
+# — they require booting a real Measure module and so don't fit the inner
+# go-test bench loop.
+
+set -euo pipefail
+
+ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "${ROOT_DIR}"
+
+COUNT="${COUNT:-5}"
+BENCHTIME="${BENCHTIME:-2s}"
+PKG="./pkg/query/vectorized/measure"
+TIMESTAMP="$(date -u +%Y%m%d-%H%M%S)"
+OUT_DIR="${ROOT_DIR}/dist/bench"
+REPORT="${OUT_DIR}/vectorized-${TIMESTAMP}.md"
+RAW="${OUT_DIR}/vectorized-${TIMESTAMP}.txt"
+GATES_LOG="${OUT_DIR}/vectorized-gates-${TIMESTAMP}.log"
+
+mkdir -p "${OUT_DIR}"
+
+echo "G5a bench runner"
+echo " pkg: ${PKG}"
+echo " count: ${COUNT}"
+echo " benchtime: ${BENCHTIME}"
+echo " output: ${REPORT}"
+echo
+
+echo "==> Running paired microbenchmarks..."
+go test "${PKG}" \
+ -run='^$' \
+ -bench='^Benchmark(RowPath|VectorizedPath)_W[1-5]$' \
+ -benchmem \
+ -count="${COUNT}" \
+ -benchtime="${BENCHTIME}" \
+ -timeout=30m \
+ | tee "${RAW}"
+
+echo
+echo "==> Running gate enforcement test (RUN_BENCH_GATES=1)..."
+gate_status=0
+RUN_BENCH_GATES=1 go test "${PKG}" \
+ -run='^TestBenchGates_PerWorkload$' \
+ -count=1 \
+ -timeout=30m \
+ -v \
+ | tee "${GATES_LOG}" || gate_status=$?
+
+echo
+echo "==> Writing report ${REPORT}"
+
+commit="$(git rev-parse --short HEAD 2>/dev/null || echo unknown)"
+host="$(uname -mnsr 2>/dev/null || echo unknown)"
+go_version="$(go version 2>/dev/null || echo unknown)"
+
+{
+ echo "# Vectorized Query Path — Microbench Report"
+ echo
+ echo "- Generated: \`${TIMESTAMP}\` UTC"
+ echo "- Commit: \`${commit}\`"
+ echo "- Host: \`${host}\`"
+ echo "- Go: \`${go_version}\`"
+ echo "- Count: ${COUNT}"
+ echo "- Benchtime: ${BENCHTIME}"
+ echo
+ echo "## Paired benchmark output"
+ echo
+ echo '```'
+ cat "${RAW}"
+ echo '```'
+ echo
+ echo "## Gate enforcement"
+ echo
+ if [ "${gate_status}" -eq 0 ]; then
+ echo "**PASS** — all gates met."
+ else
+ echo "**FAIL** — at least one gate violated. See log below."
+ fi
+ echo
+ echo '```'
+ cat "${GATES_LOG}"
+ echo '```'
+} > "${REPORT}"
+
+echo
+if [ "${gate_status}" -ne 0 ]; then
+ echo "GATE VIOLATION — see ${REPORT}"
+ exit "${gate_status}"
+fi
+
+echo "All gates passed. Report: ${REPORT}"
diff --git a/test/integration/standalone/benchmark/benchmark_suite_test.go b/test/integration/standalone/benchmark/benchmark_suite_test.go
new file mode 100644
index 000000000..d7b252d98
--- /dev/null
+++ b/test/integration/standalone/benchmark/benchmark_suite_test.go
@@ -0,0 +1,42 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package benchmark_test
+
+import (
+ "testing"
+
+ . "github.com/onsi/ginkgo/v2"
+ . "github.com/onsi/gomega"
+
+ integration_standalone "github.com/apache/skywalking-banyandb/test/integration/standalone"
+)
+
+// TestVectorizedBenchmark is the Ginkgo entry point for the macro suite. It
+// runs the W1..W5 workloads against two real Measure modules (flag-off and
+// flag-on) and asserts the spec's gate ratios on wall-clock latency, B/op,
+// and allocs/op. Default-skipped because each workload boots a full
+// standalone — set RUN_VECTORIZED_BENCH=1 to enable.
+//
+// Use:
+//
+// RUN_VECTORIZED_BENCH=1 go test ./test/integration/standalone/benchmark/...
+// --label-filter '!slow' --vv -timeout=60m
+func TestVectorizedBenchmark(t *testing.T) {
+ RegisterFailHandler(Fail)
+ RunSpecs(t, "Vectorized Macro Benchmark Suite", Label(integration_standalone.Labels...))
+}
diff --git a/test/integration/standalone/benchmark/benchmark_test.go b/test/integration/standalone/benchmark/benchmark_test.go
new file mode 100644
index 000000000..ee6cf286a
--- /dev/null
+++ b/test/integration/standalone/benchmark/benchmark_test.go
@@ -0,0 +1,63 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package benchmark_test contains the macro benchmark suite for the
+// vectorized query path. Each workload boots a real standalone Measure
+// module twice (flag-off and flag-on), drives identical queries against
+// each, and asserts the spec's gate ratios. Skipped unless
+// RUN_VECTORIZED_BENCH=1 is set so it does not run on every PR.
+package benchmark_test
+
+import (
+ "os"
+
+ "github.com/onsi/ginkgo/v2"
+)
+
+// G5a macro suite — currently a structural placeholder. The microbench
+// gate at pkg/query/vectorized/measure/bench_gates_test.go is the
+// authoritative G5a check; this suite captures end-to-end query latency
+// against a live standalone for the rollout decision and is deferred until
+// the vectorized adapter has wired operators (BatchLimit / BatchGroupBy /
+// BatchAggregation / BatchTop) into the live Pipeline. Without that
+// wiring, end-to-end W3/W4 measurements still flow through the row path's
+// post-aggregation stage and don't reflect operator-level cost.
+//
+// To enable the suite (e.g. once operator wiring lands):
+//
+// RUN_VECTORIZED_BENCH=1 go test \
+// ./test/integration/standalone/benchmark/... \
+// -timeout=60m -ginkgo.label-filter '!slow'
+var _ = ginkgo.Describe("vectorized macro benchmarks", ginkgo.Ordered, func() {
+ ginkgo.BeforeAll(func() {
+ if os.Getenv("RUN_VECTORIZED_BENCH") != "1" {
+ ginkgo.Skip("set RUN_VECTORIZED_BENCH=1 to run the macro vectorized benchmarks")
+ }
+ })
+
+ for _, w := range Workloads() {
+ w := w
+ ginkgo.It("workload "+w.ID, ginkgo.Label("slow"), func() {
+ // TODO(G5a-followup): boot two standalones (flag-off, flag-on)
+ // reusing setup.ClosableStandalone, run RunWorkload(w) against
+ // each, write the markdown report via Report(), then assert
+ // gate ratios. Tracked in the deferred items list at the
+ // bottom of .omc/plans/autopilot-impl.md.
+ ginkgo.Skip("macro implementation deferred; see microbench gate for G5a signal")
+ })
+ }
+})
diff --git a/test/integration/standalone/benchmark/workloads_test.go b/test/integration/standalone/benchmark/workloads_test.go
new file mode 100644
index 000000000..18242567b
--- /dev/null
+++ b/test/integration/standalone/benchmark/workloads_test.go
@@ -0,0 +1,60 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Package benchmark_test holds the macro benchmark workload catalog and
+// helpers. Workloads mirror the catalog in
+// .omc/specs/deep-interview-vectorized-query-critique.md §"Performance
+// Evaluation Plan". Each entry is the single source of truth shared between
+// the microbench gate (pkg/query/vectorized/measure/bench_test.go) and this
+// macro suite.
+package benchmark_test
+
+// Workload describes one macro shape — series count, points per series, and
+// the kind of work that dominates (scan, group-by, top-N).
+type Workload struct {
+ ID string
+ Description string
+ Series int
+ PointsEach int
+ Kind WorkloadKind
+}
+
+// WorkloadKind tags the benchmark intent so the runner can pick the right
+// query template per shape.
+type WorkloadKind int
+
+// Recognized workload kinds.
+const (
+ WorkloadScan WorkloadKind = iota
+ WorkloadMixedProjection
+ WorkloadGroupByAggregation
+ WorkloadTopN
+ WorkloadLargeFanout
+)
+
+// Workloads returns the W1..W5 catalog at the spec's nominal scale. The
+// macro suite scales these to fit available host capacity; the microbench
+// uses smaller variants in pkg/query/vectorized/measure/bench_test.go.
+func Workloads() []Workload {
+ return []Workload{
+ {ID: "W1", Description: "Single-series scan; 1×10 000", Series: 1, PointsEach: 10000, Kind: WorkloadScan},
+ {ID: "W2", Description: "Mixed projection; 100×1 000", Series: 100, PointsEach: 1000, Kind: WorkloadMixedProjection},
+ {ID: "W3", Description: "GroupBy + SUM/COUNT; 1 000×100", Series: 1000, PointsEach: 100, Kind: WorkloadGroupByAggregation},
+ {ID: "W4", Description: "Top-N=100 over 100×10 000", Series: 100, PointsEach: 10000, Kind: WorkloadTopN},
+ {ID: "W5", Description: "Wide fanout; 10 000×1 000 full projection", Series: 10000, PointsEach: 1000, Kind: WorkloadLargeFanout},
+ }
+}