You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by ha...@apache.org on 2026/10/20 00:10:35 UTC
(skywalking-banyandb) branch main updated: perf(measure): batch v2 and scan path v2 for the vectorized measure query, plus 0.12.0 designs (#1412)
This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
The following commit(s) were added to refs/heads/main by this push:
new 15635181b perf(measure): batch v2 and scan path v2 for the vectorized measure query, plus 0.12.0 designs (#1412)
15635181b is described below
commit 15635181bbf804750994b9d4b13d473373890de9
Author: Gao Hongtao <ha...@gmail.com>
AuthorDate: Tue Oct 20 08:10:29 2026 +0800
perf(measure): batch v2 and scan path v2 for the vectorized measure query, plus 0.12.0 designs (#1412)
* perf(bydbql): resolve schemas from the liaison's watch-fed cache
BydbQL's transformer read measure, stream and trace schemas through the
schema registry, which broadcasts to every metadata node and may repair
lagging ones on every request. The liaison already holds these schemas in
its entity repositories, the same view the native query handlers use for
gating and time-range clamping. Read them from there and fall back to the
registry only on a miss (a just-created resource whose watch event has not
arrived yet).
A loopback benchmark measured the registry read at 0.4-0.6 ms per group,
against 3-6 us for the whole transform once the schema is local.
* perf(measure): borrow decoded block memory in vectorized query batches
First part of the measure batch v2 design: stop allocating one heap object
per decoded value on the vectorized query path.
- Typed numeric decode: null-free int64/float64 columns read for a query
decode straight into column-owned []int64/[]float64 and fill batch
columns in bulk, instead of one []byte per value converted back. Merge,
flush, TopN and columns with nulls keep the byte form.
- Reference-counted block cursors: string, bytes and string-array tag
cells borrow the cursor's decoded buffers zero-copy. Batches pin their
owners (vectorized.Owners on RecordBatch and MeasureBatch); a cursor
returns to its pool only when the last borrowing batch is released.
Group-by, aggregation, TopN and the response edge copy what they keep.
BYDB_POISON_RELEASED_CURSOR=1 overwrites released cursor memory so a
use-after-release fails tests visibly.
- Native projected columns: every projected tag and field with a mapped
schema type is a native typed column rather than a protobuf
passthrough, and each response value is built in one allocation. The
wire format is unchanged (array columns are still sent as TagValue
cells).
- Aggregation and TopN no longer allocate per input row: keys are looked
up from a reused buffer and copied only on insert; a displacing TopN
row reuses the root's storage.
- The G9e gates now decode real stored parts inside the timer
(banyand/measure TestVectorizedStorageGates), compare against the
previous passthrough column selection, and pin absolute allocation
budgets. The old gates built protobufs outside the timer and are
removed.
On real stored blocks, across the eight gate workloads: geomean -6% time
and -75% allocations, with group-by + aggregation at -97% allocations and
a distributed full-scan frame -49% time and -99.7% allocations.
* perf(measure): stream the vectorized scan one series window at a time
Step 4a of scan path v2. The batch scan groups cursors into series
windows and cuts each window into segments from block metadata (a block
starting at or before the running segment end joins it). A one-cursor
segment is decoded when the scan reaches it and lent to the pipeline
without copying: string cells already borrow, and numeric, timestamp and
version columns now borrow the cursor's typed slices through
TypedColumn.Borrow, which copies before any write and drops the borrowed
array on Reset/pool release. BatchSourceFromBatchResult passes such
batches through at block size via a reference-counted holder instead of
copying them into 1024-row batches; batches under 256 rows are still
coalesced. Descending order sets a reversed Selection.
Overlapping segments, and time-ordered queries whose series overlap in
time, still use the existing merge; step 4c replaces it. TopN and
index-mode results are unchanged. Group-by reserves memory per batchSize
chunk now that input batches can reach 8,192 rows, and time-bucket
flooring copies before writing.
Scan counters (total_scan_windows, total_scan_windows_overlapped,
total_scan_segments_fast, total_scan_segments_overlap) measure how often
the overlap path is taken on a real cluster.
* perf(measure): merge time-ordered vectorized scans keys-first with lazy decode
Step 4c of scan path v2. Time-ordered scans whose series overlap in time
(the default no-ORDER-BY case) and overlapping segments inside a series
no longer decode every matching block up front and merge row by row.
timeMerge orders cursors by their first timestamp (last, for descending)
and decodes a cursor only when the merge frontier reaches it, so decoded
memory is bounded by the blocks overlapping the current instant. Each page
first merges (timestamp, series, version) keys with the legacy Less order
and newest-version dedup, then gathers every column once: numerics by
index copy, strings/bytes/string arrays as borrowed views pinned on the
batch, other cells copied as before. Finished cursors are released after
the page is gathered. Series windows disjoint from all others keep the
zero-copy lend path; TopN and index mode are unchanged. Decoding goes
through a single cursorLoader hook for the step 4b decode pool.
The merge never closes a page inside a duplicate group, fixing the legacy
batch merge emitting a straddling duplicate twice at its 4,096-row cut.
* perf(measure): decode scan blocks on a node-wide pool with reserved prefetch
Step 4b of scan path v2.
- decodePool: one per data node or standalone server, GOMAXPROCS
workers, created in PreRun and closed in GracefulStop before the
schema repository. A decode the scan is waiting on goes to an urgent
lane, and a prefetched decode that becomes the head is promoted there.
- prefetcher: windowScan and timeMerge request decodes in exactly the
order they consume them and collect them head-first, so completion
order never changes output. Depth starts at 2 and grows to 8 while
reservations succeed (inside a merge, max(8, frontier/4), at most 32).
- Reservations: each cursor reserves its uncompressed size before decode
and releases it when its refcount reaches zero, after every borrowing
batch is released. Prefetch is refused beyond the protector's headroom.
The current cursor never blocks on the protector once the query holds
memory (it waits only at admission), which rules out two queries
waiting for memory the other holds; the per-query quota keeps the
"block scan quota exceeded" error. TopN and the legacy path keep the
whole-result check, deferred to the first pull.
- Cancellation withdraws queued decodes and waits for running ones before
a cursor is released, and finished tasks drop their cursor references.
Review fixes (codex): cancel releasing a cursor whose prefetched decode
was running; protector hold-and-wait; finished tasks retaining cursors;
starvation of a prefetched head behind urgent tasks; a test that could
hang depending on worker scheduling.
* test(measure): leak-check scan resources and add a measure soak
Step 4d of scan path v2, with the fixes it surfaced.
- Leak checks: verifyNoScanLeaks fails a test that ends with block
cursors, owner pins, query results, scan reservations or decode tasks
still outstanding, and TestMains check the same at the end of the
banyand/measure and pkg/query/vectorized binaries
(ScanResourceSnapshot, OutstandingOwnerPins). The first run found 385
leaked cursors and leaked pins, all in tests that never released their
batches or built cursors outside the pool; those tests are fixed.
- Measure soak: TestMeasureVecSoak (DQB_SOAK=1) runs an in-process
two-data-node cluster with background writes, nine query shapes,
concurrent and cancelled queries, and gates on resources returning to
baseline, heap growth and sampled correctness. It runs with plain
go test; the docker path selects it with DQB_ENGINE=measure.
- Audits: selection-order and large-batch tests across the measure
operators (7 to 65,536 rows, reversed, shuffled and subset selections).
- Data races: measure queries wrote the block cache and shard ID into
parts shared with concurrent queries and decode workers. Both are now
carried in a partRef next to each part and removed from part.
- Metrics: opening a group closed the previous group's query metrics, so
a node with several groups exported only the last one's. Each group now
keeps its own set.
- blockCursor.reset clears shardID.
* fix(storage): do not panic creating a segment whose directory is pending delete
Retention removes an expired segment from the controller's list at once,
but its directory is deleted only when the last reference is released.
A write to the same time range in that window found no segment, called
create, and MkdirPanicIfExist crashed the node with "directory is exist".
The stream test data hits it by writing six days back into a group with a
three-day TTL.
Track removed segments whose directory still exists and return
ErrSegmentDeleting from create until it is gone; callers already handle
CreateSegmentIfNotExist errors. The race is pre-existing on main.
TestCreateSegment_WhileRemovedSegmentStillHeld reproduces the panic on the
old code.
* test(querybench): settle heap readings in the measure and trace soaks
The soaks compared HeapInuse after a single GC, which moved by 15% or
more from span fragmentation and pool victim caches alone, so the heap
gate failed on unchanged code. Both soaks now read live heap (HeapAlloc)
as the minimum of five samples after four GC rounds, the measure soak
warms the process-wide zstd encoders before its baseline, and the growth
percentage is taken of the workload heap. A deliberate leak of one
response in 2,000 still fails the gate (128%); twelve clean runs grew
1.0-1.6%.
* perf(measure): constant entity-tag columns and run-based aggregation
Step 5 of the batch v2 / scan path v2 designs.
- ConstColumn (pkg/query/vectorized) holds one value and a length and
reports its logical ColumnType, so schemas and planners are unchanged.
ColumnDef.Entity marks entity tags. lendBatch and single-series
coalesced batches turn projected entity tags into consts decoded from
row 0; interleaved merge pages stay typed. Every column consumer handles
consts, specialized or through Expand/Materialize; frame encoding
expands them, so the wire format is unchanged.
- Batch hints (series ends, time order within each series run) are set
only when a storage batch is lent and cleared on Reset.
- Runs: BatchAggregation (All/Map) and BatchTimeBucket fold a run of rows
whose group key is constant (consts, entity tags of a known series,
time buckets found by galloping search, asc or desc) with one group
lookup and typed range loops that add in the same order as the per-row
fold. Count-distinct, reduce mode, unknown hints and fragmented batches
(runs under three rows on average) use the per-row path.
- SelectRows / BatchColumnFilter evaluate a predicate on a const column
once per batch.
Gate workloads: geomean -25%; group-by + SUM/COUNT -67%, group by entity
tags -68%, raw group-by -61%, scalar reduce -57%, time bucket + entity on
disjoint series -39%. New gates W13-W15.
* perf(measure): columnar frame v2 between data nodes and the liaison
Step 6 of the batch v2 design.
- Frame version 4 (pkg/query/vectorized/frame/columnar.go): per column a
validity bitmap (1 = valid, only when there are nulls), int64/float64
data, or int32 offsets plus values, each zero-padded to 8 bytes,
little-endian. Constant columns are sent as one value, bounded by a
256 MiB frame-wide expansion budget that the encoder respects by
flattening. Arrays stay TagValue proto cells. Borrowed strings are
copied into the frame; the selection is applied at encode.
- Decode slices the received message: numeric columns borrow typed
slices (copied once into an aligned buffer when the message is not
8-byte aligned, or on big-endian), strings are views into the message,
which the batch pins through a MessageOwner. Every length and offset is
validated; truncation and 30,000 random mutations error without panics.
- Negotiation: the liaison sets InternalQueryRequest.max_frame_version
(additive field 3); the data node emits version 4 only when asked and
allowed (BANYAND_MEASURE_FRAME_MAX_VERSION). Frames self-describe, so
every old/new pairing works. The distributed vectorized suite now runs
three clusters: new+new, old liaison + new data nodes, new liaison +
old data nodes.
- Liaison: every consumer clones what it keeps and releases frames after
use; the frame-decode tracing probe runs only for traced queries.
appendColumnRangeCoerced now clones borrowed strings it wraps into
TagValue/FieldValue cells (found under release poisoning).
4 columns x 8,192 rows: decode 0.85 ms / 16,416 allocs -> 0.14 ms / 38,
encode 1.3 -> 0.43 ms, frame +6%. Distributed full scan -38%.
* refactor(measure)!: drop frame version negotiation and the vectorized flags
BanyanDB 0.12.0 upgrades are whole-cluster (stop every node, start the
new binary); rolling and mixed-version clusters are not supported. The
compatibility machinery added for frame v2 is therefore removed:
- InternalQueryRequest.max_frame_version, BANYAND_MEASURE_FRAME_MAX_VERSION
and the advertise/emit caps are gone. Measure data nodes always send the
columnar frame (wire version 4) and the liaison accepts only version 4,
rejecting others with an error that names the upgrade rule. The measure
row frame (version 3) encode/decode is deleted; the hidden-tag, sorted
and traced paths that used it emit the columnar frame. Stream and trace
frames are unchanged.
- --measure-vectorized-enabled, --stream-vectorized-enabled and
--trace-vectorized-enabled are removed (they were deprecated no-ops
that failed on =false); passing one is now an unknown flag.
- The three-cluster mixed-version integration setup and the version
matrix tests are removed.
- docs/operation/upgrade.md states the 0.12 whole-cluster rule and drops
the liaison-first and flag-fallback guidance; CHANGES.md marks both as
breaking.
* style(measure): name unused test parameters _ for revive
* docs(design): add the measure batch v2 and scan path v2 designs
Two linked design documents for the vectorized measure query path, now
implemented: the data unit (reference-counted cursors, borrowed typed
columns, constant columns, the columnar data-node-to-liaison frame) and
the scan that feeds it (series windows, the time-ordered merge, the
node-wide decode pool with reserved prefetch).
Each keeps the original rationale and design-time evidence and adds an
"As built" section with what shipped, where it deviates from the
proposal (no offsets or list column types, no frame version negotiation
because 0.12.0 supports whole-cluster upgrades only, SIMD and radix sort
deferred) and the measured results.
* docs(design): add the streaming API, measure QL gaps, BydbQL plan cache and aggregation designs
Add four 0.12.0 design documents and list them in both design indexes:
- Streaming API: a server-streaming StreamQuery per model, the existing
Write reworked to acknowledge per sealed batch, and an internal
cluster.v1 Scan RPC, with the memory invariant, the decisions taken and
the test and benchmark plan. Accepted, not yet implemented.
- Measure QL gaps: what the measure query language lacks against
InfluxDB 3.0 SQL and the order to close it. Design only.
- BydbQL plan cache: proto-free measure planning with a reusable plan
cache. Step 3, the local schema source, ships in this release; the
rest is not implemented.
- Aggregating measures and materialized views: an outline.
* docs(design): scope the streaming design to the existing Query and Write APIs for 0.12.0
0.12.0 implements the streaming process behind the existing APIs: the
internal Scan transport, chunked emission on data nodes, an incremental
liaison merge that the unary Query RPCs collect with today's exact dedup,
and the Write rework. Fixed per-node query deadlines become idle deadlines
with heartbeats so long scans can finish. The public StreamQuery API, its
gateway route, the streaming bydbctl options and BydbQL streaming move to
the next milestone.
* test(measure): keep the race build of the scan differential tests within the CI budget
The banyand/measure Go tests took 21.6 minutes under `go test -race
-covermode atomic` (19 minutes in CI), which pushed the "Test Banyand"
job past its 30-minute limit. Under the race build the differential
tests now run a deterministic covering selection and smaller fixtures;
the build without race keeps today's full matrices and sizes.
Tests (race seconds, before -> after): TestScanWindowConstAndRunsMatchPerRowAndLegacy
706 -> 34, TestTimeMergeMatchesLegacyThroughGroupByAggregationAndTimeBuckets
145 -> 14, TestTimeMergeMatchesLegacyAcrossGeneratedQueries 91 -> 14,
TestScanWindowMatchesLegacyForTimeOrders 62 -> 5,
TestScanWindowRawQueriesMatchWithConstColumns 42 -> 7,
TestScanWindowMatchesLegacyAcrossGeneratedQueries 25 -> 9,
TestTimeMergeDecodesCursorsAtTheFrontier 17 -> 7,
TestPrefetchAddsAtMostAQuarterOfTheFrontierToTheMergePeak 16 -> 7,
TestScanSequence.../TestPrefetchMatchesSynchronousDecode.../
TestConcurrentQueriesShareTheNodePool trimmed. Package total 21.6 min -> 4.3 min.
Coverage is preserved by coverPairs (scan_cover_test.go), a greedy,
deterministic cover over the valid cross product: every value of every
dimension appears, and so does every pair that names a relationship
(grouping x aggregation, grouping x order for the const test; every pair
for the merge group-by, time-order and raw-query tests; order, projection
and limit pairs for the generated merge queries). Cheaper combos are
preferred through a cost hint. Fixtures shrink only as far as their
comments allow: the oap fixture keeps more than ten series, the frontier
tests keep 16 series, and the mixed fixture's long series is 8,400 rows
(still several blocks, a tail part overlapping its last block, and the
4,096-row page cut). A raceEnabled constant comes from two build-tagged
test files. No test is deleted, skipped or weakened, and no product code
changes.
---
CHANGES.md | 17 +-
banyand/internal/storage/segment.go | 31 +-
banyand/internal/storage/segment_test.go | 34 +
banyand/liaison/grpc/bydbql_schema_source.go | 47 +
banyand/liaison/grpc/server.go | 7 +-
banyand/measure/benchmark_encode_test.go | 2 +-
banyand/measure/block.go | 181 ++-
banyand/measure/block_batch.go | 455 ++++++-
banyand/measure/block_batch_benchmark_test.go | 21 +-
banyand/measure/block_test.go | 4 +-
banyand/measure/cache_benchmark_test.go | 4 +-
banyand/measure/column.go | 145 ++-
banyand/measure/column_test.go | 4 +-
banyand/measure/column_typed_benchmark_test.go | 86 ++
banyand/measure/column_typed_test.go | 397 ++++++
banyand/measure/const_verify_test.go | 42 +
banyand/measure/cursor_borrow_benchmark_test.go | 86 ++
banyand/measure/cursor_borrow_test.go | 292 +++++
banyand/measure/leak_check_selftest_test.go | 94 ++
banyand/measure/leak_check_test.go | 107 ++
banyand/measure/measure.go | 67 +-
banyand/measure/metadata.go | 47 +-
banyand/measure/metrics.go | 9 +-
banyand/measure/migration_copy.go | 2 +-
.../measure/native_projection_benchmark_test.go | 355 ++++++
banyand/measure/native_projection_test.go | 344 ++++++
banyand/measure/part.go | 2 -
banyand/measure/part_iter.go | 8 +-
banyand/measure/part_iter_test.go | 6 +-
banyand/measure/part_query_race_test.go | 77 ++
banyand/measure/poison_test.go | 23 +
banyand/measure/query.go | 82 +-
banyand/measure/query_batch.go | 33 +-
banyand/measure/query_batch_top_test.go | 7 +-
banyand/measure/query_test.go | 12 +-
banyand/measure/race_off_test.go | 29 +
banyand/measure/race_on_test.go | 30 +
banyand/measure/scan_concurrent_benchmark_test.go | 62 +
banyand/measure/scan_const_test.go | 438 +++++++
banyand/measure/scan_cover_test.go | 86 ++
banyand/measure/scan_decode_pool.go | 396 ++++++
banyand/measure/scan_merge.go | 842 +++++++++++++
banyand/measure/scan_merge_test.go | 688 +++++++++++
banyand/measure/scan_metrics_test.go | 122 ++
banyand/measure/scan_prefetch.go | 334 +++++
banyand/measure/scan_prefetch_test.go | 1115 +++++++++++++++++
banyand/measure/scan_resources.go | 83 ++
banyand/measure/scan_window.go | 642 ++++++++++
banyand/measure/scan_window_benchmark_test.go | 85 ++
banyand/measure/scan_window_fixture_test.go | 379 ++++++
banyand/measure/scan_window_test.go | 769 ++++++++++++
banyand/measure/snapshot.go | 16 +-
banyand/measure/snapshot_test.go | 11 +-
banyand/measure/svc_data.go | 6 +
banyand/measure/svc_standalone.go | 5 +
banyand/measure/topn_post_processor_test.go | 3 +
banyand/measure/tracing.go | 6 +-
banyand/measure/tstable.go | 6 +-
banyand/measure/tstable_test.go | 2 +-
banyand/measure/vectorized_gates_benchmark_test.go | 764 ++++++++++++
banyand/stream/stream.go | 2 -
banyand/stream/vectorized_flag_test.go | 30 +-
banyand/trace/svc_liaison.go | 5 +-
banyand/trace/svc_standalone.go | 5 +-
banyand/trace/trace.go | 9 +-
banyand/trace/vectorized_flag_test.go | 41 -
docs/design/0.12.0/README.md | 6 +
docs/design/0.12.0/bydbql-plan-cache/README.md | 441 +++++++
.../design/0.12.0/measure-aggregation-mv/README.md | 101 ++
docs/design/0.12.0/measure-batch-v2/README.md | 646 ++++++++++
docs/design/0.12.0/measure-ql-gaps/README.md | 101 ++
docs/design/0.12.0/measure-scan-path-v2/README.md | 488 ++++++++
docs/design/0.12.0/streaming-api/README.md | 1296 ++++++++++++++++++++
docs/design/README.md | 6 +
docs/operation/upgrade.md | 40 +-
docs/soak/g5d-phase-d-summary.md | 6 +-
docs/soak/g5d-runbook.md | 6 +-
docs/soak/g9f-runbook.md | 6 +-
pkg/bydbql/schema_source.go | 75 ++
pkg/bydbql/schema_source_test.go | 164 +++
pkg/bydbql/transformer.go | 9 +-
pkg/query/aggregation/aggregation.go | 11 +
pkg/query/aggregation/function.go | 89 ++
pkg/query/aggregation/range_fold_test.go | 117 ++
pkg/query/model/batch.go | 148 ++-
pkg/query/model/batch_const_test.go | 67 +
pkg/query/model/batch_owners_test.go | 52 +
pkg/query/model/batch_selection_test.go | 75 ++
pkg/query/vectorized/appendrange.go | 19 +-
pkg/query/vectorized/batch.go | 148 ++-
pkg/query/vectorized/batch_lent_test.go | 82 ++
pkg/query/vectorized/column.go | 43 +-
pkg/query/vectorized/column_pool.go | 5 +
pkg/query/vectorized/const_column.go | 304 +++++
pkg/query/vectorized/const_column_test.go | 230 ++++
pkg/query/vectorized/frame/columnar.go | 511 ++++++++
pkg/query/vectorized/frame/columnar_decode.go | 478 ++++++++
pkg/query/vectorized/frame/doc.go | 5 +
pkg/query/vectorized/frame/encode.go | 5 +
pkg/query/vectorized/frame/frame.go | 6 +
pkg/query/vectorized/frame/golden_test.go | 18 +-
pkg/query/vectorized/frame/owner.go | 115 ++
pkg/query/vectorized/frame/validate.go | 18 +-
pkg/query/vectorized/leak_main_test.go | 29 +
pkg/query/vectorized/measure/aggregation.go | 86 +-
.../measure/aggregation_keyalloc_test.go | 183 +++
pkg/query/vectorized/measure/aggregation_reduce.go | 12 +-
pkg/query/vectorized/measure/aggregation_runs.go | 274 +++++
pkg/query/vectorized/measure/batchsource.go | 206 +++-
.../vectorized/measure/batchsource_lend_test.go | 279 +++++
pkg/query/vectorized/measure/bench_gates_test.go | 146 ---
pkg/query/vectorized/measure/bench_test.go | 788 ------------
pkg/query/vectorized/measure/bench_tracing_test.go | 7 +
.../vectorized/measure/borrowed_input_test.go | 44 +
.../vectorized/measure/const_consumers_test.go | 275 +++++
pkg/query/vectorized/measure/drain_frame_test.go | 184 +++
pkg/query/vectorized/measure/filter.go | 63 +
.../measure/frame/columnar_bench_test.go | 105 ++
.../vectorized/measure/frame/columnar_test.go | 657 ++++++++++
pkg/query/vectorized/measure/frame/decode_test.go | 46 +-
pkg/query/vectorized/measure/frame/frame.go | 65 +-
pkg/query/vectorized/measure/frame/frame_test.go | 152 +--
.../vectorized/measure/frame/leak_main_test.go | 30 +
pkg/query/vectorized/measure/groupby.go | 150 ++-
pkg/query/vectorized/measure/groupby_block_test.go | 48 +
pkg/query/vectorized/measure/integration.go | 100 +-
pkg/query/vectorized/measure/leak_main_test.go | 30 +
pkg/query/vectorized/measure/lifetime_test.go | 400 ++++++
pkg/query/vectorized/measure/plan/distributed.go | 42 +-
.../measure/plan/distributed_execute_test.go | 381 ++++++
.../vectorized/measure/plan/distributed_rows.go | 49 +-
.../measure/plan/distributed_rows_test.go | 21 +-
.../vectorized/measure/plan/leak_main_test.go | 36 +
.../vectorized/measure/plan/multi_group_schema.go | 2 +-
pkg/query/vectorized/measure/plan/sort_key.go | 8 +
pkg/query/vectorized/measure/plan/sort_key_test.go | 26 +
...ade_test.go => tracing_partial_failure_test.go} | 73 +-
.../vectorized/measure/plan/tracing_summary.go | 14 +-
pkg/query/vectorized/measure/raw_emit.go | 138 ++-
pkg/query/vectorized/measure/raw_emit_test.go | 2 +
pkg/query/vectorized/measure/reduce.go | 9 +
pkg/query/vectorized/measure/runs.go | 159 +++
.../vectorized/measure/runs_equivalence_test.go | 606 +++++++++
pkg/query/vectorized/measure/schema_native_test.go | 380 ++++++
.../vectorized/measure/selection_order_test.go | 645 ++++++++++
pkg/query/vectorized/measure/serialize.go | 218 +++-
pkg/query/vectorized/measure/timebucket.go | 13 +-
pkg/query/vectorized/measure/timebucket_runs.go | 116 ++
pkg/query/vectorized/measure/top.go | 56 +-
pkg/query/vectorized/owner.go | 119 ++
pkg/query/vectorized/owner_test.go | 103 ++
pkg/query/vectorized/pipeline.go | 7 +-
pkg/query/vectorized/pool.go | 15 +-
pkg/query/vectorized/retained_bytes.go | 5 +
pkg/query/vectorized/schema.go | 6 +
pkg/query/vectorized/select.go | 52 +
pkg/query/vectorized/selection_size_test.go | 191 +++
pkg/query/vectorized/stream/config.go | 19 +-
pkg/query/vectorized/stream/config_test.go | 17 +-
pkg/query/vectorized/typed_column.go | 103 +-
pkg/query/vectorized/typed_column_borrow_test.go | 118 ++
pkg/query/vectorized/typed_column_validity_test.go | 64 +
pkg/test/leakcheck/leakcheck.go | 114 ++
scripts/bench-vectorized.sh | 26 +-
.../distributed/query/vectorized_test.go | 6 +
.../distributed/querybench/benchmark_test.go | 5 +
test/integration/distributed/querybench/config.go | 91 +-
.../distributed/querybench/measure_soak_test.go | 973 +++++++++++++++
.../distributed/querybench/orchestrate.sh | 8 +-
.../distributed/querybench/run-docker.sh | 4 +
.../distributed/querybench/soak_test.go | 77 +-
.../standalone/benchmark/benchmark_test.go | 7 +-
172 files changed, 23791 insertions(+), 1770 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index a06fbb363..7db1d3adc 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -18,17 +18,32 @@ Release Notes.
- Bound the native index's memory: merges stream to disk instead of materializing the merged segment, so their memory no longer grows with segment size or fan-in; persisted segments are no longer held in the heap but served from their files through `pkg/fs` (one descriptor per segment, term dictionaries read in place, a per-field bloom filter of at most 2 MiB kept for existence checks); and staged merge outputs and external segments are published by rename, so no staged name survives a r [...]
- Add logging related flags to the FODC proxy and agent. Every BanyanDB binary now shares the same logging flags and the matching `BYDB_LOGGING_*` environment variables.
-- [Breaking Change] Remove the row-based query execution path from the Stream, Measure and Trace query engines. Measure TopN pre-aggregation is unchanged and still executes row-based. The `--stream-vectorized-enabled`, `--trace-vectorized-enabled` and `--measure-vectorized-enabled` flags stay registered but no longer select an engine; `=false` now fails fast at startup on standalone, data and liaison nodes. A query shape the vectorized engine cannot plan now returns an error instead of f [...]
+- [Breaking Change] Remove the row-based query execution path from the Stream, Measure and Trace query engines. Measure TopN pre-aggregation is unchanged and still executes row-based. The `--stream-vectorized-enabled`, `--trace-vectorized-enabled` and `--measure-vectorized-enabled` flags are removed; a node started with one of them fails at startup with an unknown-flag error. A query shape the vectorized engine cannot plan now returns an error instead of falling through to row execution. [...]
- Push the criteria tag filter ahead of the vectorized stream merge, so a filtered index-order query bounds its merge at limit+offset. For such a query, where several rows share an ElementID, the criteria is now evaluated first and the element is represented by its first matching row in the requested sort order. Filtered timestamp-order queries are unchanged.
- Keep Stream filtering and client projection on the columnar batches until output, so a filtered distributed Stream query emits a native columnar frame instead of per-element protobuf. A filtered index-order query no longer re-evaluates the criteria per element: the pre-merge columnar filter already decided it, and only the hidden-tag removal remains. A filtered timestamp-order query evaluates the criteria on the batches at the position its element filter held, after the merge cap, so i [...]
- `measure.v1.QueryRequest.Aggregation` can now target a tag (`tag_name` + `tag_family`) instead of a field: `SUM`/`MIN`/`MAX`/`MEAN` over an `INT` tag, and `COUNT` over any scalar tag, in both standalone and distributed queries. Wire-additive only — existing clients that only ever set `field_name` are unaffected. `COUNT_DISTINCT` is not yet implemented; see the [tag aggregation and time bucketing design](docs/design/0.12.0/tag-aggregation/README.md).
- `measure.v1.QueryRequest.GroupBy.time_bucket` groups a measure query by tumbling time windows, alone or combined with a tag `GroupBy`, in both standalone and distributed queries. The bucket width comes from the request or falls back to the measure's own `interval`. The part-scan path streams bucket-by-bucket so live memory never exceeds one bucket's groups; index-mode measures (no ascending-timestamp guarantee) use a bounded non-streaming fallback instead. A mixed-version cluster where [...]
- Improve Canopy text readability: raise the faint and dim text contrast, lift every font size to at least 12px, use heavier sub-navigation weights, stop forcing antialiased font smoothing, and load IBM Plex Mono 600 and Sans 700. The query builder rail is wider by default (400px) to fit the larger text.
- `databasev1.Node` now reports its supported file format versions and IANA time zone name.
+- Resolve BydbQL measure, stream and trace schemas from the liaison's watch-fed schema cache instead of the schema registry, which queried every metadata node on each request. A registry read is now made only when the cache has not yet seen a newly created resource. A BydbQL query no longer pays a metadata round trip per group (about 0.4–0.6 ms each on loopback), and no longer triggers the registry's read-repair of lagging metadata nodes.
+- Decode null-free int64 and float64 measure columns straight into typed slices on the vectorized query path, instead of one heap-allocated byte slice per value that the batch then converted back. Filling an 8,192-row numeric column drops from about 0.5 ms and 8,196 allocations to 40–86 µs and 2 allocations. Merge, flush, TopN and columns containing nulls keep the byte form.
+- Reference-count measure block cursors so batch string, bytes and string-array tag cells borrow the decoded block memory instead of copying each value; a cursor returns to its pool only when the last batch borrowing from it is released. Operators that keep values past their input batch (group-by, aggregation, TopN) and the response edge copy what they retain. Decoding an 8,192-row string column drops from 8,193 allocations to 1, and 33–58% in time. `BYDB_POISON_RELEASED_CURSOR=1` overwr [...]
+- Emit every projected measure tag and field with a known schema type as a native typed column instead of a protobuf passthrough column, and build each response value in one allocation. Plain scans now decode without per-value wrappers and pay them only for rows that reach the response: on real stored blocks a full scan is 16% faster with 38% fewer allocations, limited and TopN scans 23–65% faster, and a distributed full-scan frame 49% faster with 99.7% fewer allocations. The wire format [...]
+- Stop allocating per input row in measure aggregation and TopN: group and replica-dedup keys are looked up from a reused buffer and copied only when a new group is inserted, and a TopN row that displaces the heap root reuses the root's storage. On real stored blocks, group-by + aggregation queries drop from about 103k to 2.8k allocations and TopN from 472k to 32k (15% faster).
+- Stream vectorized measure scans one series window at a time instead of decoding every matching block before the first row. A series whose blocks do not overlap in time is handed to the pipeline block by block without copying, numeric and timestamp columns included, at block size (up to 8,192 rows); descending order uses a reversed selection instead of a reversed copy, and windows under 256 rows are coalesced. Overlapping segments still merge. On real stored blocks the eight gate worklo [...]
+- Replace the eager decode-and-merge of time-ordered vectorized measure scans (the default when a query has no ORDER BY) with a time-ordered merge: a block is decoded only when the merge reaches its first timestamp and released as soon as its rows are consumed, keys are merged before values, and each column is gathered once per page with strings borrowed zero-copy. Output order is unchanged; peak decoded blocks drop from every matching block to about one per series. Multi-series queries [...]
+- Decode vectorized measure scan blocks on a node-wide pool sized to GOMAXPROCS instead of one goroutine per block, prefetching ahead in consumption order with a depth that grows while memory reservations succeed. Each block reserves its decoded size before decoding and releases it when the last batch borrowing from it is released, so a scan's memory is accounted by what it holds rather than checked once against its whole result: prefetch waits under memory pressure, the protector is con [...]
+- Store a series' entity tags once per batch as a constant column in single-series vectorized measure batches, and aggregate by runs: when every group key is constant over a run of rows (constant columns, entity tags of a known series, or a time bucket over time-ordered rows), the group is looked up once per run and the run is folded with typed loops in the same order as before. Group-by + aggregation over real stored blocks is 57–68% faster; results are bit-identical.
+- [Breaking Change] Add measure frame version 4, a columnar data-node-to-liaison wire format that replaces the row-oriented measure frame (version 3): each column's validity, offsets and values are sent as contiguous 8-byte-padded buffers, and the liaison builds columns by slicing the received message instead of allocating per cell. Data nodes always send version 4 and a liaison accepts only version 4 for measure frames, rejecting any other version with an error; there is no version nego [...]
- Add the data export dry-run protocol (`transfer.v1.ExportService`) and `bydbctl data export --dry-run`.
### Bug Fixes
+- Fix the liaison's distributed measure row merge wrapping borrowed frame strings into protobuf tag and field values without copying them.
+- Fix a data node panicking with "directory is exist" when a write landed in a segment that retention had just removed but whose directory was still held by a reader; creating a segment over a directory pending deletion now returns `ErrSegmentDeleting`.
+- Fix data races in measure queries, which wrote the block cache and shard ID into parts shared with concurrent queries; both are now carried alongside each part.
+- Fix a data node with several measure groups exporting query metrics only for the most recently opened group: opening a group closed every other group's query metrics. Each group now keeps its own.
+- Fix the vectorized measure batch merge emitting a duplicate row when a same-series, same-timestamp duplicate straddled its 4,096-row page boundary; the row path already returned the deduplicated result.
- Fix stream queries silently dropping the later blocks of a series: the block filter's verdict is per-block, but `findBlock` advanced a whole series on skip.
- Fix a data race in Measure TopN where a pooled TopN value reused for a query wrote its tag names into the streaming processor's entity tag name slice.
- Stop `pkg/index/native`'s compaction-persistence test from flaking on `Test Pkg`: it asserted that garbage collection succeeds the instant a read view closes, without accounting for a background persist still holding an older root.
diff --git a/banyand/internal/storage/segment.go b/banyand/internal/storage/segment.go
index 322f8fdca..bf8f1f70f 100644
--- a/banyand/internal/storage/segment.go
+++ b/banyand/internal/storage/segment.go
@@ -48,6 +48,10 @@ var (
// ErrInvalidSegmentTimestamp is returned when a segment timestamp is zero or near-epoch,
// indicating a corrupted timestamp (e.g., unset MinTimestamp flowing through sync paths).
ErrInvalidSegmentTimestamp = errors.New("invalid segment timestamp: epoch or near-epoch time is not valid for APM data")
+ // ErrSegmentDeleting is returned when a segment is requested while a previous segment
+ // with the same directory is still being deleted: it left the controller's list but its
+ // directory is removed only once the last reference is released. The caller may retry.
+ ErrSegmentDeleting = errors.New("segment is being deleted")
)
var _ Cache = (*segmentCache)(nil)
@@ -84,7 +88,9 @@ type segment[T TSTable, O any] struct {
mu sync.RWMutex
refCount int32
mustBeDeleted uint32
- id segmentID
+ // deleted is set once performDelete has removed the directory.
+ deleted atomic.Bool
+ id segmentID
// closeFailed records that the most recent closeResourcesLocked call
// could not cleanly close the series index (the owner's final
// persistence flush reported an error). The segment itself stays usable
@@ -383,6 +389,7 @@ func (s *segment[T, O]) performDelete() {
}
s.closeResourcesLocked()
s.lfs.MustRMAll(s.location)
+ s.deleted.Store(true)
}
// delete flags the segment for deletion. If it is dormant (no active reference)
@@ -667,6 +674,9 @@ type segmentController[T TSTable, O any] struct {
opts *TSDBOpts[T, O]
l *logger.Logger
indexMetrics *idxmetrics.Metrics
+ // deleting holds segments that left lst but whose directory is not removed yet,
+ // keyed by directory. It is guarded by the embedded lock.
+ deleting map[string]*segment[T, O]
*groupCache
lfs banyanfs.FileSystem
position common.Position
@@ -1037,6 +1047,14 @@ func (sc *segmentController[T, O]) create(ctx context.Context, start time.Time)
end = stdEnd
}
segPath := path.Join(sc.location, fmt.Sprintf(segTemplate, sc.format(start)))
+ // A removed segment keeps its directory until its last reference is released.
+ // Creating over it would hit an existing directory, so ask the caller to retry.
+ if old, ok := sc.deleting[segPath]; ok {
+ if !old.deleted.Load() {
+ return nil, fmt.Errorf("%w: %s", ErrSegmentDeleting, segPath)
+ }
+ delete(sc.deleting, segPath)
+ }
sc.lfs.MkdirPanicIfExist(segPath, DirPerm)
meta := SegmentMetadata{
Version: currentVersion,
@@ -1166,8 +1184,19 @@ func (sc *segmentController[T, O]) deleteExpiredSegments(segmentSuffixes []strin
}
func (sc *segmentController[T, O]) removeSeg(segID segmentID) {
+ for loc, old := range sc.deleting {
+ if old.deleted.Load() {
+ delete(sc.deleting, loc)
+ }
+ }
for i, b := range sc.lst {
if b.id == segID {
+ if atomic.LoadUint32(&b.mustBeDeleted) != 0 && !b.deleted.Load() {
+ if sc.deleting == nil {
+ sc.deleting = make(map[string]*segment[T, O])
+ }
+ sc.deleting[b.location] = b
+ }
sc.lst = append(sc.lst[:i], sc.lst[i+1:]...)
break
}
diff --git a/banyand/internal/storage/segment_test.go b/banyand/internal/storage/segment_test.go
index 176562832..2b790003a 100644
--- a/banyand/internal/storage/segment_test.go
+++ b/banyand/internal/storage/segment_test.go
@@ -1688,3 +1688,37 @@ func TestFormatSegmentTime_RoundTrip(t *testing.T) {
require.NoError(t, err)
require.True(t, gotHour.Equal(hourT), "hour round-trip: got %s want %s", gotHour, hourT)
}
+
+// TestCreateSegment_WhileRemovedSegmentStillHeld pins the retention-versus-write race:
+// a writer holds a reference on a segment that retention then removes. The segment
+// leaves the list at once but its directory stays until the last reference is released,
+// so a write for the same time range must not panic on the existing directory. It must
+// get ErrSegmentDeleting, and succeed once the directory is gone.
+func TestCreateSegment_WhileRemovedSegmentStillHeld(t *testing.T) {
+ sc, _, cleanup := newAlignmentTestController(t, IntervalRule{Unit: DAY, Num: 1})
+ defer cleanup()
+
+ ts := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
+ seg, err := sc.createSegment(ts) // returned with a reference, like a writer holds
+ require.NoError(t, err)
+ segLocation := seg.Location()
+ require.DirExists(t, segLocation)
+
+ hasSegment, err := sc.remove(t.Context(), time.Date(2026, 10, 13, 0, 0, 0, 0, time.UTC))
+ require.NoError(t, err)
+ require.True(t, hasSegment)
+ require.DirExists(t, segLocation, "the held segment's directory must outlive the removal")
+
+ require.NotPanics(t, func() {
+ _, createErr := sc.createSegment(ts)
+ require.ErrorIs(t, createErr, ErrSegmentDeleting)
+ })
+
+ seg.DecRef() // the last reference runs the deferred delete
+ require.NoDirExists(t, segLocation)
+
+ recreated, err := sc.createSegment(ts)
+ require.NoError(t, err)
+ require.DirExists(t, recreated.Location())
+ recreated.DecRef()
+}
diff --git a/banyand/liaison/grpc/bydbql_schema_source.go b/banyand/liaison/grpc/bydbql_schema_source.go
new file mode 100644
index 000000000..053e12f8f
--- /dev/null
+++ b/banyand/liaison/grpc/bydbql_schema_source.go
@@ -0,0 +1,47 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package grpc
+
+import (
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ "github.com/apache/skywalking-banyandb/pkg/bydbql"
+)
+
+// entityRepoSchemaSource serves BydbQL schema lookups from the liaison's watch-fed
+// entity repositories, the same view the native query handlers already read for
+// gating and time-range clamping. Each kind has its own repository because each
+// discovery service registers a separate handler.
+type entityRepoSchemaSource struct {
+ measures *entityRepo
+ streams *entityRepo
+ traces *entityRepo
+}
+
+var _ bydbql.SchemaSource = (*entityRepoSchemaSource)(nil)
+
+func (s *entityRepoSchemaSource) Measure(group, name string) (*databasev1.Measure, bool) {
+ return s.measures.getMeasure(identity{name: name, group: group})
+}
+
+func (s *entityRepoSchemaSource) Stream(group, name string) (*databasev1.Stream, bool) {
+ return s.streams.getStream(identity{name: name, group: group})
+}
+
+func (s *entityRepoSchemaSource) Trace(group, name string) (*databasev1.Trace, bool) {
+ return s.traces.getTrace(identity{name: name, group: group})
+}
diff --git a/banyand/liaison/grpc/server.go b/banyand/liaison/grpc/server.go
index 80407f918..ff9be2759 100644
--- a/banyand/liaison/grpc/server.go
+++ b/banyand/liaison/grpc/server.go
@@ -189,9 +189,14 @@ func NewServer(_ context.Context, tir1Client, tir2Client, broadcaster queue.Clie
discoveryService: newDiscoveryService(schema.KindProperty, schemaRegistry, nr.PropertyNodeRegistry, gr),
queryBudget: protector.QueryBudgetFor(protectorService),
}
+ bydbQLSchemaSource := &entityRepoSchemaSource{
+ measures: measureSVC.entityRepo,
+ streams: streamSVC.entityRepo,
+ traces: traceSVC.entityRepo,
+ }
bydbQLSVC := &bydbQLService{
repo: schemaRegistry,
- transformer: bydbql.NewTransformer(schemaRegistry),
+ transformer: bydbql.NewTransformerWithSource(schemaRegistry, bydbQLSchemaSource),
streamSvc: streamSVC,
measureSvc: measureSVC,
traceSvc: traceSVC,
diff --git a/banyand/measure/benchmark_encode_test.go b/banyand/measure/benchmark_encode_test.go
index 973424237..968b99f68 100644
--- a/banyand/measure/benchmark_encode_test.go
+++ b/banyand/measure/benchmark_encode_test.go
@@ -95,7 +95,7 @@ func benchmarkColumnEncoding[T any](
unmarshaled := &column{}
decoder := &encoding.BytesBlockDecoder{}
- unmarshaled.mustReadValues(decoder, buf, *cm, uint64(size))
+ unmarshaled.mustReadValues(decoder, buf, *cm, uint64(size), false)
b.StopTimer()
totalDecodeTimeNs += time.Since(startDecode).Nanoseconds()
diff --git a/banyand/measure/block.go b/banyand/measure/block.go
index 3164d076f..056768e91 100644
--- a/banyand/measure/block.go
+++ b/banyand/measure/block.go
@@ -19,8 +19,11 @@ package measure
import (
"fmt"
+ "math"
+ "os"
"slices"
"sort"
+ "sync/atomic"
"github.com/apache/skywalking-banyandb/api/common"
databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
@@ -216,7 +219,7 @@ func (b *block) marshalTagFamily(tf columnFamily, bm *blockMetadata, ww *writers
func (b *block) unmarshalTagFamily(decoder *encoding.BytesBlockDecoder, tfIndex int, name string,
columnFamilyMetadataBlock *dataBlock, tagProjection []string, metaReader, valueReader fs.Reader, count int,
- schemaTagTypes map[string]pbv1.ValueType,
+ schemaTagTypes map[string]pbv1.ValueType, typedNumeric bool,
) {
if len(tagProjection) < 1 {
return
@@ -241,7 +244,7 @@ NEXT:
typedName := encodeTypedColumn(tp, schemaType)
for i := range cfm.columnMetadata {
if cfm.columnMetadata[i].name == typedName {
- cc[j].mustReadValues(decoder, valueReader, cfm.columnMetadata[i], uint64(b.Len()))
+ cc[j].mustReadValues(decoder, valueReader, cfm.columnMetadata[i], uint64(b.Len()), typedNumeric)
cc[j].name = tp
continue NEXT
}
@@ -252,7 +255,7 @@ NEXT:
if hasSchemaType && cfm.columnMetadata[i].valueType != schemaType {
continue
}
- cc[j].mustReadValues(decoder, valueReader, cfm.columnMetadata[i], uint64(b.Len()))
+ cc[j].mustReadValues(decoder, valueReader, cfm.columnMetadata[i], uint64(b.Len()), typedNumeric)
continue NEXT
}
}
@@ -300,11 +303,7 @@ func (b *block) uncompressedSizeBytes() uint64 {
n += uint64(len(tf.name))
for _, c := range tf.columns {
n += uint64(len(c.name))
- for _, v := range c.values {
- if len(v) > 0 {
- n += uint64(len(v))
- }
- }
+ n += c.bytesSize()
}
}
@@ -312,23 +311,33 @@ func (b *block) uncompressedSizeBytes() uint64 {
for i := range ff.columns {
c := ff.columns[i]
nameLen := uint64(len(c.name))
- for _, v := range c.values {
- if len(v) > 0 {
- n += nameLen + uint64(len(v))
+ switch c.form {
+ case numericFormInt64, numericFormFloat64:
+ n += uint64(c.rowCount()) * (nameLen + 8)
+ default:
+ for _, v := range c.values {
+ if len(v) > 0 {
+ n += nameLen + uint64(len(v))
+ }
}
}
}
return n
}
-func (b *block) mustReadFrom(decoder *encoding.BytesBlockDecoder, p *part, bm blockMetadata, schemaTagTypes map[string]pbv1.ValueType) {
+// mustReadFrom loads the block bm describes. typedNumeric lets null-free
+// int64/float64 columns load in their typed form (see column.mustReadValues);
+// callers must pass true only if every reader of the block understands it.
+func (b *block) mustReadFrom(decoder *encoding.BytesBlockDecoder, p *part, bm blockMetadata,
+ schemaTagTypes map[string]pbv1.ValueType, typedNumeric bool,
+) {
b.reset()
b.timestamps, b.versions = mustReadTimestampsFrom(b.timestamps, b.versions, &bm.timestamps, int(bm.count), p.timestamps)
cc := b.field.resizeColumns(len(bm.field.columnMetadata))
for i := range cc {
- cc[i].mustReadValues(decoder, p.fieldValues, bm.field.columnMetadata[i], bm.count)
+ cc[i].mustReadValues(decoder, p.fieldValues, bm.field.columnMetadata[i], bm.count, typedNumeric)
}
_ = b.resizeTagFamilies(len(bm.tagProjection))
@@ -350,7 +359,7 @@ func (b *block) mustReadFrom(decoder *encoding.BytesBlockDecoder, p *part, bm bl
}
b.unmarshalTagFamily(decoder, i, name, block,
bm.tagProjection[i].Names, p.tagFamilyMetadata[name],
- p.tagFamilies[name], int(bm.count), schemaTagTypes)
+ p.tagFamilies[name], int(bm.count), schemaTagTypes, typedNumeric)
}
}
@@ -457,8 +466,19 @@ func releaseBlock(b *block) {
var blockPool = pool.Register[*block]("measure-block")
+// blockCursor is one decoded block of one series, read by a query.
+//
+// It is reference-counted. The query holds one reference from generateBlockCursor
+// until it calls releaseBlockCursor; every batch whose string, bytes or
+// string-array cells borrow from the cursor's decoded buffers (see
+// borrowTagRows) holds one more through vectorized.Owners. The cursor is reset
+// and pooled only when the count reaches zero, so a borrowed cell stays valid
+// until the last batch that holds it is released.
type blockCursor struct {
- p *part
+ p *part
+ // res is the scan's memory reservation for this cursor, released when the
+ // last reference goes (see releaseBlockCursor). It is nil outside a window scan.
+ res *scanReservation
fields columnFamily
timestamps []int64
versions []int64
@@ -471,12 +491,22 @@ type blockCursor struct {
idx int
minTimestamp int64
maxTimestamp int64
+ refs atomic.Int32
shardID common.ShardID
}
+// Retain adds one reference. The caller must already hold one.
+func (bc *blockCursor) Retain() { bc.refs.Add(1) }
+
+// Release drops one reference; at zero the cursor is recycled. It implements
+// vectorized.Owner.
+func (bc *blockCursor) Release() { releaseBlockCursor(bc) }
+
func (bc *blockCursor) reset() {
bc.idx = 0
bc.p = nil
+ bc.res = nil
+ bc.shardID = 0
bc.bm.reset()
bc.minTimestamp = 0
bc.maxTimestamp = 0
@@ -815,7 +845,11 @@ func (bc *blockCursor) replace(r *model.MeasureResult, storedIndexValue map[comm
}
}
-func (bc *blockCursor) loadData(tmpBlock *block) bool {
+// loadData reads the cursor's block into tmpBlock and copies the rows inside the
+// query time range into bc. typedNumeric keeps null-free int64/float64 columns in
+// their typed form; pass true only when every reader of bc is a batch reader
+// that understands column.form (copyAllToBatch, copyToBatch, replaceInBatch).
+func (bc *blockCursor) loadData(tmpBlock *block, typedNumeric bool) bool {
tmpBlock.reset()
cfm := make([]columnMetadata, 0, len(bc.fieldProjection))
NEXT_FIELD:
@@ -845,7 +879,7 @@ NEXT_FIELD:
}
}
bc.bm.tagFamilies = tf
- tmpBlock.mustReadFrom(&bc.columnValuesDecoder, bc.p, bc.bm, bc.schemaTagTypes)
+ tmpBlock.mustReadFrom(&bc.columnValuesDecoder, bc.p, bc.bm, bc.schemaTagTypes, typedNumeric)
start, end, ok := timestamp.FindRange(tmpBlock.timestamps, bc.minTimestamp, bc.maxTimestamp)
if !ok {
@@ -863,32 +897,32 @@ NEXT_FIELD:
name: cf.columns[i].name,
valueType: cf.columns[i].valueType,
}
- if len(cf.columns[i].values) == 0 {
+ if cf.columns[i].rowCount() == 0 {
continue
}
- if len(cf.columns[i].values) != len(tmpBlock.timestamps) {
- logger.Panicf("unexpected number of values for tags %q: got %d; want %d", cf.columns[i].name, len(cf.columns[i].values), len(tmpBlock.timestamps))
+ if cf.columns[i].rowCount() != len(tmpBlock.timestamps) {
+ logger.Panicf("unexpected number of values for tags %q: got %d; want %d", cf.columns[i].name, cf.columns[i].rowCount(), len(tmpBlock.timestamps))
}
- column.values = append(column.values, cf.columns[i].values[start:end+1]...)
+ column.copyRowsFrom(&cf.columns[i], start, end+1)
tf.columns = append(tf.columns, column)
}
bc.tagFamilies = append(bc.tagFamilies, tf)
}
bc.fields.name = tmpBlock.field.name
for i := range tmpBlock.field.columns {
- if len(tmpBlock.field.columns[i].values) == 0 {
+ if tmpBlock.field.columns[i].rowCount() == 0 {
continue
}
- if len(tmpBlock.field.columns[i].values) != len(tmpBlock.timestamps) {
+ if tmpBlock.field.columns[i].rowCount() != len(tmpBlock.timestamps) {
logger.Panicf("unexpected number of values for fields %q: got %d; want %d",
- tmpBlock.field.columns[i].name, len(tmpBlock.field.columns[i].values), len(tmpBlock.timestamps))
+ tmpBlock.field.columns[i].name, tmpBlock.field.columns[i].rowCount(), len(tmpBlock.timestamps))
}
c := column{
name: tmpBlock.field.columns[i].name,
valueType: tmpBlock.field.columns[i].valueType,
}
- c.values = append(c.values, tmpBlock.field.columns[i].values[start:end+1]...)
+ c.copyRowsFrom(&tmpBlock.field.columns[i], start, end+1)
bc.fields.columns = append(bc.fields.columns, c)
}
return true
@@ -896,19 +930,114 @@ NEXT_FIELD:
var blockCursorPool = pool.Register[*blockCursor]("measure-blockCursor")
+// poisonReleasedCursor makes a cursor overwrite every decoded value it owns with
+// poisonByte when its last reference is dropped, so a borrowed cell read after
+// release returns visibly wrong data, and makes a release beyond the last
+// reference panic. It is a test aid, enabled by BYDB_POISON_RELEASED_CURSOR=1 or
+// setPoisonReleasedCursor; it costs one pass over the decoded bytes per cursor.
+var poisonReleasedCursor atomic.Bool
+
+// poisonByte is the sentinel written over released cursor memory. It is not valid
+// UTF-8 on its own and is unlikely to appear in test data.
+const poisonByte = 0xDD
+
+func init() {
+ poisonReleasedCursor.Store(os.Getenv("BYDB_POISON_RELEASED_CURSOR") == "1")
+}
+
+// setPoisonReleasedCursor switches poisoning and returns the previous setting.
+func setPoisonReleasedCursor(on bool) bool { return poisonReleasedCursor.Swap(on) }
+
+// generateBlockCursor returns a cursor holding exactly one reference.
func generateBlockCursor() *blockCursor {
v := blockCursorPool.Get()
if v == nil {
- return &blockCursor{}
+ v = &blockCursor{}
}
+ v.refs.Store(1)
return v
}
+// releaseBlockCursor drops one reference. The cursor is reset and returned to the
+// pool only when the last reference goes; until then batches that borrow its
+// decoded buffers keep reading valid data. A release with no reference left is a
+// bug: it panics under poisoning and is otherwise ignored, because pooling the
+// cursor a second time would hand one cursor to two owners.
func releaseBlockCursor(bc *blockCursor) {
+ n := bc.refs.Add(-1)
+ if n > 0 {
+ return
+ }
+ if n < 0 {
+ bc.refs.Add(1)
+ if poisonReleasedCursor.Load() {
+ logger.Panicf("blockCursor released more times than it was retained")
+ }
+ return
+ }
+ if poisonReleasedCursor.Load() {
+ bc.poison()
+ }
+ // Nothing borrows the decoded buffer any more; let the next block reuse it.
+ bc.columnValuesDecoder.Reset()
+ if res := bc.res; res != nil {
+ bc.res = nil
+ res.release()
+ }
bc.reset()
blockCursorPool.Put(bc)
}
+// poison overwrites the memory every decoded value of bc points at. Values alias
+// the decoder buffer (plain encoding) or a block-sized allocation made by the
+// dictionary decoder; overwriting through the values reaches both, and also any
+// array the decoder buffer outgrew while it was filled.
+//
+// Typed numeric columns, timestamps and versions are lent to batches as views of
+// the cursor's slices (blockCursor.lendBatch), so they are overwritten too: a
+// batch that outlives its cursor reads poisonNumeric instead of its values.
+func (bc *blockCursor) poison() {
+ for i := range bc.tagFamilies {
+ for j := range bc.tagFamilies[i].columns {
+ bc.tagFamilies[i].columns[j].poisonNumeric()
+ poisonValues(bc.tagFamilies[i].columns[j].values)
+ }
+ }
+ for i := range bc.fields.columns {
+ bc.fields.columns[i].poisonNumeric()
+ poisonValues(bc.fields.columns[i].values)
+ }
+ for i := range bc.timestamps {
+ bc.timestamps[i] = poisonNumeric
+ }
+ for i := range bc.versions {
+ bc.versions[i] = poisonNumeric
+ }
+}
+
+// poisonNumeric is the sentinel written over released numeric cursor memory: -1
+// as an int64 (all 0xFF bytes) and a NaN as a float64, neither of which a test
+// fixture writes.
+const poisonNumeric = -1
+
+// poisonNumeric overwrites the typed numeric slices of c.
+func (c *column) poisonNumeric() {
+ for i := range c.int64s {
+ c.int64s[i] = poisonNumeric
+ }
+ for i := range c.float64s {
+ c.float64s[i] = math.NaN()
+ }
+}
+
+func poisonValues(values [][]byte) {
+ for _, v := range values {
+ for k := range v {
+ v[k] = poisonByte
+ }
+ }
+}
+
type blockPointer struct {
block
bm blockMetadata
diff --git a/banyand/measure/block_batch.go b/banyand/measure/block_batch.go
index afb007cf4..7cf817760 100644
--- a/banyand/measure/block_batch.go
+++ b/banyand/measure/block_batch.go
@@ -18,8 +18,12 @@
package measure
import (
+ "fmt"
+ "sync/atomic"
+
"github.com/apache/skywalking-banyandb/api/common"
modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
"github.com/apache/skywalking-banyandb/pkg/convert"
"github.com/apache/skywalking-banyandb/pkg/logger"
pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
@@ -27,6 +31,31 @@ import (
"github.com/apache/skywalking-banyandb/pkg/query/vectorized"
)
+// disableConstColumns switches the production of ConstColumns off, so a test can
+// compare a scan that carries entity tags as constants with one that does not.
+var disableConstColumns atomic.Bool
+
+// verifyEntityConstants makes every ConstColumn the scan builds be checked against
+// all the rows it stands for. Tests turn it on; production never does.
+var verifyEntityConstants atomic.Bool
+
+func constColumnsEnabled() bool { return !disableConstColumns.Load() }
+
+// assertColumnIsConst panics unless every row of col holds the value of constCol.
+func assertColumnIsConst(col vectorized.Column, constCol *vectorized.ConstColumn, what string) {
+ for i := range col.Len() {
+ rowConst, ok := vectorized.AcquireConstFrom(col, i, 1)
+ if !ok {
+ panic(fmt.Sprintf("measure: %s: cannot read row %d as a constant", what, i))
+ }
+ same := rowConst.Equal(constCol)
+ vectorized.ReleaseColumn(rowConst)
+ if !same {
+ panic(fmt.Sprintf("measure: %s: row %d differs from the constant the batch carries for it", what, i))
+ }
+ }
+}
+
// G5b/G5c — typed-column emit path on blockCursor.
//
// copyAllToBatch and copyToBatch mirror copyAllTo / copyTo (block.go) but
@@ -120,17 +149,119 @@ func (bc *blockCursor) copyAllToBatch(b *model.MeasureBatch, schema *vectorized.
case vectorized.RoleTag:
col := b.Tags[tagIdx]
tagIdx++
- bc.fillTagColumnAllRows(col, def, idx, offset, indexValue, desc)
+ bc.fillTagColumnAllRows(b, col, def, idx, offset, indexValue, desc, false)
+ case vectorized.RoleField:
+ col := b.Fields[fieldIdx]
+ fieldIdx++
+ bc.fillFieldColumnAllRows(b, col, def, idx, offset, desc, false)
+ case vectorized.RoleTimestamp, vectorized.RoleVersion,
+ vectorized.RoleSeriesID, vectorized.RoleShardID,
+ vectorized.RoleElementID, vectorized.RoleOrderKey:
+ // Metadata handled above.
+ }
+ }
+}
+
+// lendBatch builds a batch that lends the whole decoded block instead of copying
+// it: Timestamps, Versions and typed numeric columns are views of the cursor's
+// slices, string, bytes and string-array cells borrow its decoded bytes, and the
+// batch pins the cursor until it is released. Rows stay in storage order; a
+// descending scan sets Selection to [n-1, ..., 0] rather than reversing columns.
+// Hidden (index-substituted) tags, missing columns and columns with nulls are
+// filled into owned columns row by row, as copyAllToBatch does.
+//
+// The caller keeps its own reference to bc and must not hand it a block of more
+// than 65,536 rows when desc is set (a uint16 selection cannot address them).
+func (bc *blockCursor) lendBatch(schema *vectorized.BatchSchema,
+ storedIndexValue map[common.SeriesID]map[string]*modelv1.TagValue, desc bool,
+) *model.MeasureBatch {
+ size := len(bc.timestamps)
+ var b *model.MeasureBatch
+ if constColumnsEnabled() {
+ b = model.AcquireConstEntityMeasureBatch(schema, size)
+ } else {
+ b = model.AcquireBorrowingMeasureBatch(schema, size)
+ }
+ b.BorrowMetadata(bc.timestamps, bc.versions)
+ b.Owners.Pin(bc)
+ sid := bc.bm.seriesID
+ for range size {
+ b.ShardIDs = append(b.ShardIDs, bc.shardID)
+ b.SeriesIDs = append(b.SeriesIDs, sid)
+ }
+ var indexValue map[string]*modelv1.TagValue
+ if storedIndexValue != nil {
+ indexValue = storedIndexValue[sid]
+ }
+ tagIdx, fieldIdx := 0, 0
+ for _, def := range schema.Columns {
+ switch def.Role {
+ case vectorized.RoleTag:
+ col := b.Tags[tagIdx]
+ if def.Entity && size > 0 && constColumnsEnabled() && bc.constEntityTag(b, tagIdx, def, indexValue, size) {
+ tagIdx++
+ continue
+ }
+ tagIdx++
+ bc.fillTagColumnAllRows(b, col, def, 0, size, indexValue, false, true)
case vectorized.RoleField:
col := b.Fields[fieldIdx]
fieldIdx++
- bc.fillFieldColumnAllRows(col, def, idx, offset, desc)
+ bc.fillFieldColumnAllRows(b, col, def, 0, size, false, true)
case vectorized.RoleTimestamp, vectorized.RoleVersion,
vectorized.RoleSeriesID, vectorized.RoleShardID,
vectorized.RoleElementID, vectorized.RoleOrderKey:
// Metadata handled above.
}
}
+ if desc {
+ b.Selection = make([]uint16, size)
+ for i := range b.Selection {
+ b.Selection[i] = uint16(size - 1 - i)
+ }
+ }
+ return b
+}
+
+// constEntityTag replaces tag column tagIdx of b, which holds one series' rows, with
+// a ConstColumn of size rows holding the entity tag's value. An entity tag is the same
+// in every row of a block, because a block is one series and the series is the entity
+// values; the value is decoded from the first row only, through the same path that
+// fills a whole column, so a missing family or column, a stored type that differs from
+// the schema's, and an index-substituted value all come out as they would row by row
+// (a null becomes a null constant). A string or bytes value borrows the cursor's
+// memory, which b pins. It returns false, having left the column as it was, when the
+// column is not of a type a ConstColumn can stand for.
+func (bc *blockCursor) constEntityTag(b *model.MeasureBatch, tagIdx int, def vectorized.ColumnDef,
+ indexValue map[string]*modelv1.TagValue, size int,
+) bool {
+ col := b.Tags[tagIdx]
+ if !vectorized.ConstSupported(def.Type) || col.Len() != 0 {
+ return false
+ }
+ bc.fillTagColumnAllRows(b, col, def, 0, 1, indexValue, false, false)
+ constCol, ok := b.ConstFrom(col, 0, size)
+ if !ok {
+ col.Reset()
+ return false
+ }
+ if verifyEntityConstants.Load() {
+ bc.verifyConstEntityTag(b, def, constCol, indexValue, size)
+ }
+ vectorized.ReleaseColumn(col)
+ b.Tags[tagIdx] = constCol
+ return true
+}
+
+// verifyConstEntityTag panics unless every row of the block holds the value of
+// constCol. It is the test-mode check of the invariant constEntityTag relies on.
+func (bc *blockCursor) verifyConstEntityTag(b *model.MeasureBatch, def vectorized.ColumnDef, constCol *vectorized.ConstColumn,
+ indexValue map[string]*modelv1.TagValue, size int,
+) {
+ full := vectorized.AcquireColumn(def.Type, size)
+ defer vectorized.ReleaseColumn(full)
+ bc.fillTagColumnAllRows(b, full, def, 0, size, indexValue, false, false)
+ assertColumnIsConst(full, constCol, "block of series "+fmt.Sprint(bc.bm.seriesID)+" tag "+def.Name)
}
// fillTagColumnAllRows appends rows [idx, offset) of the tag described by
@@ -147,9 +278,13 @@ func (bc *blockCursor) copyAllToBatch(b *model.MeasureBatch, schema *vectorized.
// 4. Validate the column's stored valueType matches the projection's
// schemaType. On mismatch, null-fill (same defensive behavior as
// copyAllTo's hasSchemaType check).
-// 5. Decode each stored byte slice into the typed column.
-func (bc *blockCursor) fillTagColumnAllRows(col vectorized.Column, def vectorized.ColumnDef,
- idx, offset int, indexValue map[string]*modelv1.TagValue, desc bool,
+// 5. Decode each stored byte slice into the typed column. String, bytes and
+// string-array columns borrow the cursor's decoded bytes instead of copying
+// them (borrowTagRows); b then pins the cursor. When lend is set, a typed
+// numeric column is lent as a zero-copy view of the cursor's slice too
+// (lendNumericRows).
+func (bc *blockCursor) fillTagColumnAllRows(b *model.MeasureBatch, col vectorized.Column, def vectorized.ColumnDef,
+ idx, offset int, indexValue map[string]*modelv1.TagValue, desc, lend bool,
) {
size := offset - idx
// Hidden tag substitution.
@@ -175,22 +310,20 @@ func (bc *blockCursor) fillTagColumnAllRows(col vectorized.Column, def vectorize
appendNullTagN(col, def, size)
return
}
- if desc {
- for i := offset - 1; i >= idx; i-- {
- fillTagCell(col, def, column.valueType, column.values[i])
- }
- } else {
- for i := idx; i < offset; i++ {
- fillTagCell(col, def, column.valueType, column.values[i])
- }
+ if bc.borrowTagRows(b, col, def, column, idx, offset, desc) {
+ return
+ }
+ if lend && !desc && bc.lendNumericRows(b, col, def, column, idx, offset) {
+ return
}
+ fillTagRows(col, def, column, idx, offset, desc)
}
// fillFieldColumnAllRows mirrors fillTagColumnAllRows but on the field side.
// There is no indexValue path for fields and no schemaType check (fields
// don't have hidden-tag substitutions).
-func (bc *blockCursor) fillFieldColumnAllRows(col vectorized.Column, def vectorized.ColumnDef,
- idx, offset int, desc bool,
+func (bc *blockCursor) fillFieldColumnAllRows(b *model.MeasureBatch, col vectorized.Column, def vectorized.ColumnDef,
+ idx, offset int, desc, lend bool,
) {
size := offset - idx
column := bc.findFieldColumn(def.Name)
@@ -198,17 +331,259 @@ func (bc *blockCursor) fillFieldColumnAllRows(col vectorized.Column, def vectori
appendNullFieldN(col, def, size)
return
}
+ if lend && !desc && bc.lendNumericRows(b, col, def, column, idx, offset) {
+ return
+ }
+ fillFieldRows(col, def, column, idx, offset, desc)
+}
+
+// lendNumericRows makes col a zero-copy view of rows [idx, offset) of a typed
+// int64 or float64 column and pins bc on b so the slice stays valid until b is
+// released. It returns false, having changed nothing, when it does not apply:
+// the column is passthrough, not in typed form (it has nulls or was decoded to
+// bytes) or of another value type, so the caller takes the copying path.
+//
+// The view is capacity-limited and col is marked borrowed (TypedColumn.Borrow),
+// so neither an append nor a Reset can write into the cursor's slice.
+func (bc *blockCursor) lendNumericRows(b *model.MeasureBatch, col vectorized.Column, def vectorized.ColumnDef,
+ column *column, idx, offset int,
+) bool {
+ switch column.form {
+ case numericFormInt64:
+ if def.Type != vectorized.ColumnTypeInt64 {
+ return false
+ }
+ c, ok := col.(*vectorized.TypedColumn[int64])
+ if !ok || c.Len() != 0 {
+ return false
+ }
+ c.Borrow(column.int64s[idx:offset])
+ case numericFormFloat64:
+ if def.Type != vectorized.ColumnTypeFloat64 {
+ return false
+ }
+ c, ok := col.(*vectorized.TypedColumn[float64])
+ if !ok || c.Len() != 0 {
+ return false
+ }
+ c.Borrow(column.float64s[idx:offset])
+ case numericFormBytes:
+ return false
+ }
+ b.Owners.Pin(bc)
+ return true
+}
+
+// fillTagRows appends rows [idx, offset) of column onto col, reversed when desc.
+// A typed int64 column is appended in bulk; anything else goes cell by cell.
+func fillTagRows(col vectorized.Column, def vectorized.ColumnDef, column *column, idx, offset int, desc bool) {
+ if column.form == numericFormInt64 &&
+ appendNumericRows(col, def, vectorized.ColumnTypeTagValue, column.int64s[idx:offset], desc, int64TagValue) {
+ return
+ }
if desc {
for i := offset - 1; i >= idx; i-- {
- fillFieldCell(col, def, column.valueType, column.values[i])
+ fillTagCell(col, def, column.valueType, column.rowBytes(i))
}
} else {
for i := idx; i < offset; i++ {
- fillFieldCell(col, def, column.valueType, column.values[i])
+ fillTagCell(col, def, column.valueType, column.rowBytes(i))
}
}
}
+// fillFieldRows is the field-side counterpart of fillTagRows.
+func fillFieldRows(col vectorized.Column, def vectorized.ColumnDef, column *column, idx, offset int, desc bool) {
+ switch column.form {
+ case numericFormInt64:
+ if appendNumericRows(col, def, vectorized.ColumnTypeFieldValue, column.int64s[idx:offset], desc, int64FieldValue) {
+ return
+ }
+ case numericFormFloat64:
+ if appendNumericRows(col, def, vectorized.ColumnTypeFieldValue, column.float64s[idx:offset], desc, float64FieldValue) {
+ return
+ }
+ case numericFormBytes:
+ }
+ if desc {
+ for i := offset - 1; i >= idx; i-- {
+ fillFieldCell(col, def, column.valueType, column.rowBytes(i))
+ }
+ } else {
+ for i := idx; i < offset; i++ {
+ fillFieldCell(col, def, column.valueType, column.rowBytes(i))
+ }
+ }
+}
+
+// appendNumericRows copies src onto col, reversed when desc. A passthrough
+// column of def.Type == passthroughType receives wrap(v) per row, matching
+// mustDecodeTagValue/mustDecodeFieldValue; a native TypedColumn[T] receives the
+// values themselves. It returns false, having appended nothing, when col is
+// neither, so the caller can take the generic path that reports the mismatch.
+func appendNumericRows[T int64 | float64, P any](col vectorized.Column, def vectorized.ColumnDef,
+ passthroughType vectorized.ColumnType, src []T, desc bool, wrap func(T) P,
+) bool {
+ if def.Type == passthroughType {
+ c, ok := col.(*vectorized.TypedColumn[P])
+ if !ok {
+ return false
+ }
+ if desc {
+ for i := len(src) - 1; i >= 0; i-- {
+ c.Append(wrap(src[i]))
+ }
+ } else {
+ for _, v := range src {
+ c.Append(wrap(v))
+ }
+ }
+ return true
+ }
+ c, ok := col.(*vectorized.TypedColumn[T])
+ if !ok {
+ return false
+ }
+ if desc {
+ for i := len(src) - 1; i >= 0; i-- {
+ c.Append(src[i])
+ }
+ } else {
+ c.AppendSlice(src)
+ }
+ return true
+}
+
+// borrowTagRows appends rows [idx, offset) of a string, bytes or string-array tag
+// as zero-copy views over the cursor's decoded bytes, reversed when desc, and pins
+// bc on b so the bytes stay valid until b is released. It returns false, having
+// appended nothing, when it does not apply: the column is passthrough, typed
+// numeric, or of another value type, so the caller takes the copying path.
+//
+// The views are valid because every value of a bytes-form column points into
+// memory only this cursor writes and only its last release recycles: the decoder
+// buffer for plain encoding, or a block-sized allocation the dictionary decoder
+// makes for dictionary encoding (the pooled Dictionary holds just the headers).
+// Bytes cells are capacity-limited so an append cannot write into a neighbor.
+func (bc *blockCursor) borrowTagRows(b *model.MeasureBatch, col vectorized.Column, def vectorized.ColumnDef,
+ column *column, idx, offset int, desc bool,
+) bool {
+ if def.Type == vectorized.ColumnTypeTagValue || column.form != numericFormBytes {
+ return false
+ }
+ var borrowed bool
+ switch column.valueType {
+ case pbv1.ValueTypeStr:
+ c, ok := col.(*vectorized.TypedColumn[string])
+ if !ok {
+ return false
+ }
+ forEachRow(idx, offset, desc, func(i int) {
+ raw := column.values[i]
+ if raw == nil {
+ c.AppendNull()
+ return
+ }
+ borrowed = borrowed || len(raw) > 0
+ c.Append(convert.BytesToString(raw))
+ })
+ case pbv1.ValueTypeBinaryData:
+ c, ok := col.(*vectorized.TypedColumn[[]byte])
+ if !ok {
+ return false
+ }
+ forEachRow(idx, offset, desc, func(i int) {
+ raw := column.values[i]
+ if raw == nil {
+ c.AppendNull()
+ return
+ }
+ borrowed = borrowed || len(raw) > 0
+ c.Append(raw[:len(raw):len(raw)])
+ })
+ case pbv1.ValueTypeStrArr:
+ c, ok := col.(*vectorized.TypedColumn[[]string])
+ if !ok {
+ return false
+ }
+ borrowed = borrowStrArrRows(c, column, idx, offset, desc)
+ default:
+ return false
+ }
+ if borrowed {
+ b.Owners.Pin(bc)
+ }
+ return true
+}
+
+// forEachRow calls fn for each row in [idx, offset), last to first when desc.
+func forEachRow(idx, offset int, desc bool, fn func(i int)) {
+ if desc {
+ for i := offset - 1; i >= idx; i-- {
+ fn(i)
+ }
+ return
+ }
+ for i := idx; i < offset; i++ {
+ fn(i)
+ }
+}
+
+// borrowStrArrRows appends the string-array cells of rows [idx, offset) to c. An
+// element without escape sequences is a view over the stored bytes; one with
+// escapes is unescaped into a fresh string, since its decoded form is not stored.
+// All cells slice one growing slab instead of allocating a slice per row. It
+// reports whether any element borrows.
+func borrowStrArrRows(c *vectorized.TypedColumn[[]string], column *column, idx, offset int, desc bool) bool {
+ bb := bigValuePool.Generate()
+ defer bigValuePool.Release(bb)
+ var slab []string
+ var borrowed bool
+ forEachRow(idx, offset, desc, func(i int) {
+ if appendBorrowedStrArr(c, column.values[i], &slab, bb) {
+ borrowed = true
+ }
+ })
+ return borrowed
+}
+
+// appendBorrowedStrArr appends one stored string-array cell to c: null for a nil
+// raw, nil for a cell with no elements, else a capacity-limited window over slab.
+// Elements borrow raw as borrowStrArrRows describes. It reports whether any element
+// borrows. bb is scratch for unescaping.
+func appendBorrowedStrArr(c *vectorized.TypedColumn[[]string], raw []byte, slab *[]string, bb *bytes.Buffer) bool {
+ if raw == nil {
+ c.AppendNull()
+ return false
+ }
+ borrowed := false
+ start := len(*slab)
+ for buf := raw; len(buf) > 0; {
+ var rest []byte
+ var err error
+ bb.Buf, rest, err = unmarshalVarArray(bb.Buf[:0], buf)
+ if err != nil {
+ logger.Panicf("unmarshalVarArray failed: %v", err)
+ }
+ // consumed includes the trailing delimiter; unescaping only ever shrinks
+ // the element, so equal lengths mean the stored bytes are the element.
+ consumed := len(buf) - len(rest)
+ if len(bb.Buf) == consumed-1 {
+ *slab = append(*slab, convert.BytesToString(buf[:len(bb.Buf)]))
+ borrowed = borrowed || len(bb.Buf) > 0
+ } else {
+ *slab = append(*slab, string(bb.Buf))
+ }
+ buf = rest
+ }
+ if len(*slab) == start {
+ c.Append(nil)
+ return borrowed
+ }
+ c.Append((*slab)[start:len(*slab):len(*slab)])
+ return borrowed
+}
+
// fillTagCell decodes one stored cell of valueType into col, dispatching on
// def.Type so passthrough columns (ColumnTypeTagValue) keep using
// mustDecodeTagValue while native columns go through the byte-to-typed
@@ -258,7 +633,7 @@ func (bc *blockCursor) copyToBatch(b *model.MeasureBatch, schema *vectorized.Bat
case vectorized.RoleTag:
col := b.Tags[tagIdx]
tagIdx++
- bc.fillTagCell(col, def, bc.idx, indexValue)
+ bc.fillTagCell(b, col, def, bc.idx, indexValue)
case vectorized.RoleField:
col := b.Fields[fieldIdx]
fieldIdx++
@@ -272,7 +647,7 @@ func (bc *blockCursor) copyToBatch(b *model.MeasureBatch, schema *vectorized.Bat
}
// fillTagCell appends a single tag cell at row rowIdx from bc into col.
-func (bc *blockCursor) fillTagCell(col vectorized.Column, def vectorized.ColumnDef,
+func (bc *blockCursor) fillTagCell(b *model.MeasureBatch, col vectorized.Column, def vectorized.ColumnDef,
rowIdx int, indexValue map[string]*modelv1.TagValue,
) {
if indexValue != nil && indexValue[def.Name] != nil {
@@ -294,7 +669,10 @@ func (bc *blockCursor) fillTagCell(col vectorized.Column, def vectorized.ColumnD
appendNullTagN(col, def, 1)
return
}
- fillTagCell(col, def, column.valueType, column.values[rowIdx])
+ if bc.borrowTagRows(b, col, def, column, rowIdx, rowIdx+1, false) {
+ return
+ }
+ fillTagRows(col, def, column, rowIdx, rowIdx+1, false)
}
// fillFieldCell appends a single field cell at row rowIdx from bc into col.
@@ -304,7 +682,7 @@ func (bc *blockCursor) fillFieldCell(col vectorized.Column, def vectorized.Colum
appendNullFieldN(col, def, 1)
return
}
- fillFieldCell(col, def, column.valueType, column.values[rowIdx])
+ fillFieldRows(col, def, column, rowIdx, rowIdx+1, false)
}
// replaceInBatch overwrites the LAST row of b (at index lastRow) with the
@@ -369,7 +747,7 @@ func (bc *blockCursor) setTagCellAt(col vectorized.Column, def vectorized.Column
setNullTagAt(col, def, destRow)
return
}
- setTagCellAt(col, def, column.valueType, column.values[srcRow], destRow)
+ setTagCellAt(col, def, column.valueType, column.rowBytes(srcRow), destRow)
}
// setFieldCellAt overwrites column position destRow with the field value
@@ -380,7 +758,7 @@ func (bc *blockCursor) setFieldCellAt(col vectorized.Column, def vectorized.Colu
setNullFieldAt(col, def, destRow)
return
}
- setFieldCellAt(col, def, column.valueType, column.values[srcRow], destRow)
+ setFieldCellAt(col, def, column.valueType, column.rowBytes(srcRow), destRow)
}
// setTagCellAt overwrites one cell in col at destRow. Mirrors fillTagCell but
@@ -628,3 +1006,34 @@ func (bc *blockCursor) findFieldColumn(name string) *column {
}
return nil
}
+
+// constifyEntityTags replaces the entity tag columns of b, a batch of one series' rows
+// gathered from several pieces, with ConstColumns holding the first row's value. The
+// caller has established that b holds a single series. A string or bytes value keeps
+// borrowing the memory b pins.
+func constifyEntityTags(b *model.MeasureBatch, schema *vectorized.BatchSchema) {
+ n := b.RowCount()
+ if n == 0 || !constColumnsEnabled() {
+ return
+ }
+ tagIdx := 0
+ for _, def := range schema.Columns {
+ if def.Role != vectorized.RoleTag {
+ continue
+ }
+ idx := tagIdx
+ tagIdx++
+ if !def.Entity || !vectorized.ConstSupported(def.Type) {
+ continue
+ }
+ constCol, ok := b.ConstFrom(b.Tags[idx], 0, n)
+ if !ok {
+ continue
+ }
+ if verifyEntityConstants.Load() {
+ assertColumnIsConst(b.Tags[idx], constCol, "coalesced batch tag "+def.Name)
+ }
+ vectorized.ReleaseColumn(b.Tags[idx])
+ b.Tags[idx] = constCol
+ }
+}
diff --git a/banyand/measure/block_batch_benchmark_test.go b/banyand/measure/block_batch_benchmark_test.go
index 7f30abae0..c0a352105 100644
--- a/banyand/measure/block_batch_benchmark_test.go
+++ b/banyand/measure/block_batch_benchmark_test.go
@@ -71,8 +71,9 @@ var benchStorageSchemaTagTypes = map[string]pbv1.ValueType{
"env_id": pbv1.ValueTypeInt64,
}
-// benchPassthroughSchema mirrors what BuildBatchSchema currently produces
-// in production: tag/field cells stay as *modelv1.TagValue / FieldValue.
+// benchPassthroughSchema is the pre-native column selection: tag/field cells
+// stay as *modelv1.TagValue / FieldValue. BuildBatchSchema now emits native
+// columns by default; this is the baseline the native benchmarks compare to.
func benchPassthroughSchema() *vectorized.BatchSchema {
return vectorized.NewBatchSchema([]vectorized.ColumnDef{
{Role: vectorized.RoleTimestamp, Type: vectorized.ColumnTypeInt64},
@@ -138,7 +139,7 @@ func buildBenchStorageDataPoints(seriesCount, rowsPerSeries int) *dataPoints {
type benchStorageHarness struct {
tst *tsTable
snap *snapshot
- parts []*part
+ parts []partRef
sids []common.SeriesID
queryOpts queryOptions
}
@@ -163,7 +164,7 @@ func setupBenchStorageHarness(b testing.TB) *benchStorageHarness {
require.NotNil(b, snap)
shardCache := storage.NewShardCache("bench-group", 0, 0)
- parts, _ := snap.getParts(nil, shardCache, 0, int64(benchStorageRowsPer)+1)
+ parts, _ := snap.getParts(nil, shardCache, 0, 0, int64(benchStorageRowsPer)+1)
require.NotEmpty(b, parts)
sids := make([]common.SeriesID, benchStorageSeries)
@@ -190,7 +191,7 @@ func setupBenchStorageHarness(b testing.TB) *benchStorageHarness {
// drainCursors walks every block in the harness's parts and invokes f
// against each loaded cursor. Mirrors loadCursorsForBatch's per-block
// step but inline so the bench stays single-goroutine and deterministic.
-func (h *benchStorageHarness) drainCursors(f func(bc *blockCursor)) {
+func (h *benchStorageHarness) drainCursors(typedNumeric bool, f func(bc *blockCursor)) {
ti := &tstIter{}
ti.init(h.parts, h.sids, h.queryOpts.minTimestamp, h.queryOpts.maxTimestamp)
for ti.nextBlock() {
@@ -201,7 +202,7 @@ func (h *benchStorageHarness) drainCursors(f func(bc *blockCursor)) {
opts.FieldProjection = benchStorageFieldProj
bc.init(p.p, p.curBlock, opts)
tmpBlock := generateBlock()
- if bc.loadData(tmpBlock) {
+ if bc.loadData(tmpBlock, typedNumeric) {
f(bc)
}
releaseBlock(tmpBlock)
@@ -214,7 +215,7 @@ func BenchmarkStorageDecode_Row(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
- h.drainCursors(func(bc *blockCursor) {
+ h.drainCursors(false, func(bc *blockCursor) {
r := &model.MeasureResult{}
bc.copyAllTo(r, nil, benchStorageProj, false)
})
@@ -227,9 +228,10 @@ func BenchmarkStorageDecode_Passthrough(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
- h.drainCursors(func(bc *blockCursor) {
+ h.drainCursors(true, func(bc *blockCursor) {
mb := newMeasureBatchForSchema(schema, len(bc.timestamps))
bc.copyAllToBatch(mb, schema, nil, false)
+ mb.Release()
})
}
}
@@ -240,9 +242,10 @@ func BenchmarkStorageDecode_Native(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for b.Loop() {
- h.drainCursors(func(bc *blockCursor) {
+ h.drainCursors(true, func(bc *blockCursor) {
mb := newMeasureBatchForSchema(schema, len(bc.timestamps))
bc.copyAllToBatch(mb, schema, nil, false)
+ mb.Release()
})
}
}
diff --git a/banyand/measure/block_test.go b/banyand/measure/block_test.go
index 04b0bb61b..c6ef4b9e1 100644
--- a/banyand/measure/block_test.go
+++ b/banyand/measure/block_test.go
@@ -329,7 +329,7 @@ func Test_marshalAndUnmarshalTagFamily(t *testing.T) {
unmarshaled.timestamps = make([]int64, len(b.timestamps))
unmarshaled.resizeTagFamilies(1)
- unmarshaled.unmarshalTagFamily(decoder, tfIndex, name, bm.getTagFamilyMetadata(name), tagProjection[name], metaBuffer, dataBuffer, 1, nil)
+ unmarshaled.unmarshalTagFamily(decoder, tfIndex, name, bm.getTagFamilyMetadata(name), tagProjection[name], metaBuffer, dataBuffer, 1, nil, false)
if diff := cmp.Diff(unmarshaled.tagFamilies[0], b.tagFamilies[0],
cmp.AllowUnexported(columnFamily{}, column{}),
@@ -419,7 +419,7 @@ func Test_marshalAndUnmarshalBlock(t *testing.T) {
})
}
bm.tagProjection = tp
- unmarshaled.mustReadFrom(decoder, p, bm, nil)
+ unmarshaled.mustReadFrom(decoder, p, bm, nil, false)
// blockMetadata is using a map, so the order of tag families is not guaranteed
unmarshaled.sortTagFamilies()
diff --git a/banyand/measure/cache_benchmark_test.go b/banyand/measure/cache_benchmark_test.go
index 3ae9206fd..20e2c819f 100644
--- a/banyand/measure/cache_benchmark_test.go
+++ b/banyand/measure/cache_benchmark_test.go
@@ -356,7 +356,7 @@ func (m *measure) QueryWithCache(ctx context.Context, mqo model.MeasureQueryOpti
}
}()
mqo.TagProjection = newTagProjection
- var parts []*part
+ var parts []partRef
qo := queryOptions{
MeasureQueryOptions: mqo,
minTimestamp: mqo.TimeRange.Start.UnixNano(),
@@ -368,7 +368,7 @@ func (m *measure) QueryWithCache(ctx context.Context, mqo model.MeasureQueryOpti
if s == nil {
continue
}
- parts, n = s.getParts(parts, cache, qo.minTimestamp, qo.maxTimestamp)
+ parts, n = s.getParts(parts, cache, 0, qo.minTimestamp, qo.maxTimestamp)
if n < 1 {
s.decRef()
continue
diff --git a/banyand/measure/column.go b/banyand/measure/column.go
index 795110b2e..923cb0c1f 100644
--- a/banyand/measure/column.go
+++ b/banyand/measure/column.go
@@ -128,10 +128,29 @@ var (
dictionaryPool = pool.Register[*encoding.Dictionary]("measure-dictionary")
)
+// numericForm tells which in-memory representation holds a column's rows.
+type numericForm uint8
+
+const (
+ // numericFormBytes keeps every row as an encoded []byte in column.values.
+ numericFormBytes numericForm = iota
+ // numericFormInt64 keeps every row in column.int64s; column.values is empty.
+ numericFormInt64
+ // numericFormFloat64 keeps every row in column.float64s; column.values is empty.
+ numericFormFloat64
+)
+
+// column holds the rows of one tag or field. Rows live either in values (the
+// default) or, for a null-free int64/float64 column read for a query with typed
+// decoding enabled, in int64s/float64s. form says which one is populated, and
+// the other representations are always empty.
type column struct {
name string
values [][]byte
+ int64s []int64
+ float64s []float64
valueType pbv1.ValueType
+ form numericForm
}
func (c *column) reset() {
@@ -142,6 +161,71 @@ func (c *column) reset() {
values[i] = nil
}
c.values = values[:0]
+ c.clearNumeric()
+}
+
+// clearNumeric drops the typed rows, keeping their buffers for reuse.
+func (c *column) clearNumeric() {
+ c.int64s = c.int64s[:0]
+ c.float64s = c.float64s[:0]
+ c.form = numericFormBytes
+}
+
+// rowCount returns the number of rows held, whichever representation is used.
+func (c *column) rowCount() int {
+ switch c.form {
+ case numericFormInt64:
+ return len(c.int64s)
+ case numericFormFloat64:
+ return len(c.float64s)
+ default:
+ return len(c.values)
+ }
+}
+
+// bytesSize returns the total encoded size of the non-empty rows, matching what
+// summing len(v) over the bytes form gives.
+func (c *column) bytesSize() uint64 {
+ switch c.form {
+ case numericFormInt64:
+ return uint64(len(c.int64s)) * 8
+ case numericFormFloat64:
+ return uint64(len(c.float64s)) * 8
+ case numericFormBytes:
+ }
+ var n uint64
+ for _, v := range c.values {
+ n += uint64(len(v))
+ }
+ return n
+}
+
+// rowBytes returns row i in the encoded form the bytes representation holds. For
+// a typed column it re-encodes the value into a fresh slice, so it is meant for
+// single-row and fallback readers, not bulk loops.
+func (c *column) rowBytes(i int) []byte {
+ switch c.form {
+ case numericFormInt64:
+ return convert.Int64ToBytes(c.int64s[i])
+ case numericFormFloat64:
+ return convert.Float64ToBytes(c.float64s[i])
+ default:
+ return c.values[i]
+ }
+}
+
+// copyRowsFrom appends rows [start, end) of src, which must have the same form
+// as c, onto c.
+func (c *column) copyRowsFrom(src *column, start, end int) {
+ c.form = src.form
+ switch src.form {
+ case numericFormInt64:
+ c.int64s = append(c.int64s, src.int64s[start:end]...)
+ case numericFormFloat64:
+ c.float64s = append(c.float64s, src.float64s[start:end]...)
+ case numericFormBytes:
+ c.values = append(c.values, src.values[start:end]...)
+ }
}
func (c *column) resizeValues(valuesLen int) [][]byte {
@@ -277,9 +361,15 @@ func (c *column) encodeDefault(bb *bytes.Buffer) {
bb.Buf = append([]byte{byte(encoding.EncodeTypeDictionary)}, bb.Buf...)
}
-func (c *column) mustReadValues(decoder *encoding.BytesBlockDecoder, reader fs.Reader, cm columnMetadata, count uint64) {
+// mustReadValues reads one column block. When typedNumeric is true, a null-free
+// int64/float64 column is decoded straight into c.int64s/c.float64s instead of
+// per-row bytes; every other column, and every column when typedNumeric is false,
+// is decoded into c.values. Only readers that understand numericForm (the
+// vectorized query path) may pass true.
+func (c *column) mustReadValues(decoder *encoding.BytesBlockDecoder, reader fs.Reader, cm columnMetadata, count uint64, typedNumeric bool) {
c.name = cm.name
c.valueType = cm.valueType
+ c.clearNumeric()
if c.valueType == pbv1.ValueTypeUnknown {
for i := uint64(0); i < count; i++ {
c.values = append(c.values, nil)
@@ -295,12 +385,13 @@ func (c *column) mustReadValues(decoder *encoding.BytesBlockDecoder, reader fs.R
}
bb.Buf = bytes.ResizeOver(bb.Buf, int(valuesSize))
fs.MustReadData(reader, int64(cm.offset), bb.Buf)
- c.decodeColumnValues(decoder, reader.Path(), count, bb)
+ c.decodeColumnValues(decoder, reader.Path(), count, bb, typedNumeric)
}
func (c *column) mustSeqReadValues(decoder *encoding.BytesBlockDecoder, reader *seqReader, cm columnMetadata, count uint64) {
c.name = cm.name
c.valueType = cm.valueType
+ c.clearNumeric()
if cm.offset != reader.bytesRead {
logger.Panicf("%s: offset mismatch: %d vs %d", reader.Path(), cm.offset, reader.bytesRead)
}
@@ -314,26 +405,21 @@ func (c *column) mustSeqReadValues(decoder *encoding.BytesBlockDecoder, reader *
bb.Buf = bytes.ResizeOver(bb.Buf, int(valuesSize))
reader.mustReadFull(bb.Buf)
- c.decodeColumnValues(decoder, reader.Path(), count, bb)
+ c.decodeColumnValues(decoder, reader.Path(), count, bb, false)
}
-func (c *column) decodeColumnValues(decoder *encoding.BytesBlockDecoder, path string, count uint64, bb *bytes.Buffer) {
+func (c *column) decodeColumnValues(decoder *encoding.BytesBlockDecoder, path string, count uint64, bb *bytes.Buffer, typedNumeric bool) {
switch c.valueType {
case pbv1.ValueTypeInt64:
- c.decodeInt64Column(decoder, path, count, bb)
+ c.decodeInt64Column(decoder, path, count, bb, typedNumeric)
case pbv1.ValueTypeFloat64:
- c.decodeFloat64Column(decoder, path, count, bb)
+ c.decodeFloat64Column(decoder, path, count, bb, typedNumeric)
default:
c.decodeDefault(decoder, bb, count, path)
}
}
-func (c *column) decodeInt64Column(decoder *encoding.BytesBlockDecoder, path string, count uint64, bb *bytes.Buffer) {
- // decode integer type
- intValuesPtr := generateInt64Slice(int(count))
- intValues := *intValuesPtr
- defer releaseInt64Slice(intValuesPtr)
-
+func (c *column) decodeInt64Column(decoder *encoding.BytesBlockDecoder, path string, count uint64, bb *bytes.Buffer, typedNumeric bool) {
if len(bb.Buf) < 1 {
logger.Panicf("bb.Buf length too short: expect at least %d bytes, but got %d bytes", 1, len(bb.Buf))
}
@@ -351,6 +437,19 @@ func (c *column) decodeInt64Column(decoder *encoding.BytesBlockDecoder, path str
firstValue := convert.BytesToInt64(bb.Buf[1:9])
bb.Buf = bb.Buf[9:]
var err error
+ if typedNumeric {
+ // Decode straight into the column-owned slice, which survives reset().
+ c.int64s, err = encoding.BytesToInt64List(c.int64s[:0], bb.Buf, encodeType, firstValue, int(count))
+ if err != nil {
+ logger.Panicf("%s: cannot decode int values: %v", path, err)
+ }
+ c.values = c.values[:0]
+ c.form = numericFormInt64
+ return
+ }
+ intValuesPtr := generateInt64Slice(int(count))
+ intValues := *intValuesPtr
+ defer releaseInt64Slice(intValuesPtr)
intValues, err = encoding.BytesToInt64List(intValues[:0], bb.Buf, encodeType, firstValue, int(count))
if err != nil {
logger.Panicf("%s: cannot decode int values: %v", path, err)
@@ -362,14 +461,11 @@ func (c *column) decodeInt64Column(decoder *encoding.BytesBlockDecoder, path str
}
}
-func (c *column) decodeFloat64Column(decoder *encoding.BytesBlockDecoder, path string, count uint64, bb *bytes.Buffer) {
+func (c *column) decodeFloat64Column(decoder *encoding.BytesBlockDecoder, path string, count uint64, bb *bytes.Buffer, typedNumeric bool) {
// decode float type
intValuesPtr := generateInt64Slice(int(count))
intValues := *intValuesPtr
defer releaseInt64Slice(intValuesPtr)
- floatValuesPtr := generateFloat64Slice(int(count))
- floatValues := *floatValuesPtr
- defer releaseFloat64Slice(floatValuesPtr)
if len(bb.Buf) < 1 {
logger.Panicf("bb.Buf length too short: expect at least %d bytes, but got %d bytes", 1, len(bb.Buf))
@@ -393,13 +489,28 @@ func (c *column) decodeFloat64Column(decoder *encoding.BytesBlockDecoder, path s
if err != nil {
logger.Panicf("%s: cannot decode int values: %v", path, err)
}
- floatValues, err = encoding.DecimalIntListToFloat64List(floatValues[:0], intValues, exp, int(count))
+ var floatValues []float64
+ if typedNumeric {
+ // Convert straight into the column-owned slice, which survives reset().
+ floatValues = c.float64s[:0]
+ } else {
+ floatValuesPtr := generateFloat64Slice(int(count))
+ defer releaseFloat64Slice(floatValuesPtr)
+ floatValues = (*floatValuesPtr)[:0]
+ }
+ floatValues, err = encoding.DecimalIntListToFloat64List(floatValues, intValues, exp, int(count))
if err != nil {
logger.Panicf("cannot convert DecimalIntList to Float64List: %v", err)
}
if uint64(len(floatValues)) != count {
logger.Panicf("unexpected floatValues length: got %d, expected %d", len(floatValues), count)
}
+ if typedNumeric {
+ c.float64s = floatValues
+ c.values = c.values[:0]
+ c.form = numericFormFloat64
+ return
+ }
// convert float64 array to byte array
c.values = make([][]byte, count)
for i, v := range floatValues {
diff --git a/banyand/measure/column_test.go b/banyand/measure/column_test.go
index e54bd3f6a..1728062a4 100644
--- a/banyand/measure/column_test.go
+++ b/banyand/measure/column_test.go
@@ -106,7 +106,7 @@ func TestColumn_mustWriteTo_mustReadValues(t *testing.T) {
decoder := &encoding.BytesBlockDecoder{}
unmarshaled := &column{}
- unmarshaled.mustReadValues(decoder, buf, *cm, uint64(len(original.values)))
+ unmarshaled.mustReadValues(decoder, buf, *cm, uint64(len(original.values)), false)
assert.Equal(t, original.name, unmarshaled.name)
assert.Equal(t, original.valueType, unmarshaled.valueType)
@@ -245,7 +245,7 @@ func TestColumn_HighCardinalityStringEncoding(t *testing.T) {
// Test roundtrip: decode and verify all values are preserved
decoder := &encoding.BytesBlockDecoder{}
unmarshaled := &column{}
- unmarshaled.mustReadValues(decoder, buf, *cm, uint64(len(testColumn.values)))
+ unmarshaled.mustReadValues(decoder, buf, *cm, uint64(len(testColumn.values)), false)
assert.Equal(t, testColumn.name, unmarshaled.name)
assert.Equal(t, testColumn.valueType, unmarshaled.valueType)
diff --git a/banyand/measure/column_typed_benchmark_test.go b/banyand/measure/column_typed_benchmark_test.go
new file mode 100644
index 000000000..00b60f477
--- /dev/null
+++ b/banyand/measure/column_typed_benchmark_test.go
@@ -0,0 +1,86 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
+ "github.com/apache/skywalking-banyandb/pkg/encoding"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+// BenchmarkColumnDecodeFill measures what loading one 8,192-row numeric column
+// costs on the query path: decode the stored block into a pooled block column,
+// copy the rows into the cursor column (as loadData does), then fill a native
+// batch column. "bytes" is the per-row []byte form every query used before;
+// "typed" decodes straight into a typed slice.
+//
+// Run it with -bench BenchmarkColumnDecodeFill -benchmem -count=6.
+func BenchmarkColumnDecodeFill(b *testing.B) {
+ const rows = 8192
+ for _, vt := range []pbv1.ValueType{pbv1.ValueTypeInt64, pbv1.ValueTypeFloat64} {
+ typeName := "int64"
+ if vt == pbv1.ValueTypeFloat64 {
+ typeName = "float64"
+ }
+ for _, kind := range []string{"counter", "gauge"} {
+ src := &column{name: "c", valueType: vt, values: genTypedValues(vt, kind, rows, false)}
+ cm := &columnMetadata{}
+ buf := &bytes.Buffer{}
+ w := &writer{}
+ w.init(buf)
+ src.mustWriteTo(cm, w)
+ for _, typed := range []bool{false, true} {
+ mode := "bytes"
+ if typed {
+ mode = "typed"
+ }
+ b.Run(fmt.Sprintf("%s/%s/%s", typeName, kind, mode), func(b *testing.B) {
+ def := vectorized.ColumnDef{Role: vectorized.RoleField, Name: "c"}
+ var dst vectorized.Column
+ if vt == pbv1.ValueTypeInt64 {
+ def.Type = vectorized.ColumnTypeInt64
+ dst = vectorized.NewInt64Column(rows)
+ } else {
+ def.Type = vectorized.ColumnTypeFloat64
+ dst = vectorized.NewFloat64Column(rows)
+ }
+ decoder := &encoding.BytesBlockDecoder{}
+ tmp := &column{}
+ b.ReportAllocs()
+ b.ResetTimer()
+ for b.Loop() {
+ tmp.reset()
+ tmp.mustReadValues(decoder, buf, *cm, rows, typed)
+ cursorCol := column{valueType: vt}
+ cursorCol.copyRowsFrom(tmp, 0, rows)
+ dst.Reset()
+ fillFieldRows(dst, def, &cursorCol, 0, rows, false)
+ }
+ b.StopTimer()
+ require.Equal(b, rows, dst.Len())
+ })
+ }
+ }
+ }
+}
diff --git a/banyand/measure/column_typed_test.go b/banyand/measure/column_typed_test.go
new file mode 100644
index 000000000..5f9484c45
--- /dev/null
+++ b/banyand/measure/column_typed_test.go
@@ -0,0 +1,397 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "math"
+ "math/rand"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ "github.com/apache/skywalking-banyandb/pkg/encoding"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+const typedTestRows = 64
+
+// genTypedValues returns encoded rows of a counter (monotonic) or gauge
+// (noisy) series. withNull replaces one row with nil so the column falls back
+// to the bytes encoding.
+func genTypedValues(vt pbv1.ValueType, kind string, rows int, withNull bool) [][]byte {
+ r := rand.New(rand.NewSource(11)) //nolint:gosec // deterministic test data.
+ values := make([][]byte, rows)
+ cur := int64(1_000_000)
+ for i := range values {
+ switch kind {
+ case "counter":
+ cur += int64(r.Intn(50))
+ default:
+ cur = int64(r.NormFloat64()*5000) + 500_000
+ }
+ if vt == pbv1.ValueTypeFloat64 {
+ values[i] = convert.Float64ToBytes(math.Round(float64(cur)*7) / 100)
+ } else {
+ values[i] = convert.Int64ToBytes(cur)
+ }
+ }
+ if withNull {
+ values[rows/2] = nil
+ }
+ return values
+}
+
+// readBack encodes c with mustWriteTo and decodes it into dst, the way a part
+// round trip does.
+func readBack(t *testing.T, c *column, dst *column, typedNumeric bool) {
+ t.Helper()
+ cm := &columnMetadata{}
+ buf := &bytes.Buffer{}
+ w := &writer{}
+ w.init(buf)
+ c.mustWriteTo(cm, w)
+ dst.mustReadValues(&encoding.BytesBlockDecoder{}, buf, *cm, uint64(len(c.values)), typedNumeric)
+}
+
+func TestColumnTypedDecode(t *testing.T) {
+ verifyNoScanLeaks(t)
+ for _, vt := range []pbv1.ValueType{pbv1.ValueTypeInt64, pbv1.ValueTypeFloat64} {
+ for _, kind := range []string{"counter", "gauge"} {
+ for _, withNull := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%v/%s/null=%v", vt, kind, withNull), func(t *testing.T) {
+ src := &column{name: "c", valueType: vt, values: genTypedValues(vt, kind, 4096, withNull)}
+ bytesCol, typedCol := &column{}, &column{}
+ readBack(t, src, bytesCol, false)
+ readBack(t, src, typedCol, true)
+
+ require.Equal(t, numericFormBytes, bytesCol.form)
+ require.Equal(t, src.values, bytesCol.values)
+ require.Equal(t, len(src.values), typedCol.rowCount())
+ if withNull {
+ // Nulls force the plain encoding, which has no typed form.
+ require.Equal(t, numericFormBytes, typedCol.form)
+ require.Equal(t, bytesCol.values, typedCol.values)
+ return
+ }
+ require.Empty(t, typedCol.values)
+ for i := range src.values {
+ require.Equal(t, src.values[i], typedCol.rowBytes(i), "row %d", i)
+ }
+ if vt == pbv1.ValueTypeInt64 {
+ require.Equal(t, numericFormInt64, typedCol.form)
+ require.Empty(t, typedCol.float64s)
+ } else {
+ require.Equal(t, numericFormFloat64, typedCol.form)
+ require.Empty(t, typedCol.int64s)
+ for i, v := range typedCol.float64s {
+ require.Equal(t, math.Float64bits(convert.BytesToFloat64(src.values[i])), math.Float64bits(v))
+ }
+ }
+ })
+ }
+ }
+ }
+}
+
+func TestColumnTypedReuseDoesNotLeak(t *testing.T) {
+ verifyNoScanLeaks(t)
+ for _, vt := range []pbv1.ValueType{pbv1.ValueTypeInt64, pbv1.ValueTypeFloat64} {
+ t.Run(fmt.Sprint(vt), func(t *testing.T) {
+ clean := &column{name: "c", valueType: vt, values: genTypedValues(vt, "gauge", 32, false)}
+ nulls := &column{name: "c", valueType: vt, values: genTypedValues(vt, "counter", 16, true)}
+ reused := &column{}
+
+ readBack(t, clean, reused, true)
+ require.NotEqual(t, numericFormBytes, reused.form)
+ reused.reset()
+ require.Equal(t, numericFormBytes, reused.form)
+ require.Zero(t, reused.rowCount())
+ require.Empty(t, reused.int64s)
+ require.Empty(t, reused.float64s)
+
+ // typed block, then bytes-form block into the same column.
+ readBack(t, clean, reused, true)
+ readBack(t, nulls, reused, true)
+ require.Equal(t, numericFormBytes, reused.form)
+ require.Empty(t, reused.int64s)
+ require.Empty(t, reused.float64s)
+ require.Equal(t, nulls.values, reused.values)
+
+ // bytes block, then typed block into the same column.
+ readBack(t, clean, reused, true)
+ require.NotEqual(t, numericFormBytes, reused.form)
+ require.Empty(t, reused.values)
+ require.Equal(t, len(clean.values), reused.rowCount())
+ for i := range clean.values {
+ require.Equal(t, clean.values[i], reused.rowBytes(i))
+ }
+
+ // A sequential (merge) read must never produce the typed form.
+ reused.reset()
+ readBack(t, clean, reused, false)
+ require.Equal(t, numericFormBytes, reused.form)
+ require.Equal(t, clean.values, reused.values)
+ })
+ }
+}
+
+// TestColumnTypedReuseKeepsBuffer proves the typed slice is owned by the column
+// and survives reset(), so steady-state decoding does not allocate it again.
+func TestColumnTypedReuseKeepsBuffer(t *testing.T) {
+ verifyNoScanLeaks(t)
+ src := &column{name: "c", valueType: pbv1.ValueTypeInt64, values: genTypedValues(pbv1.ValueTypeInt64, "gauge", 1024, false)}
+ c := &column{}
+ readBack(t, src, c, true)
+ first := &c.int64s[0]
+ c.reset()
+ readBack(t, src, c, true)
+ require.Same(t, first, &c.int64s[0])
+}
+
+// buildTypedTestCursor builds a cursor whose columns went through a real
+// write/read round trip, in the typed form when typedNumeric is set.
+func buildTypedTestCursor(t *testing.T, kind string, withNull, typedNumeric bool) *blockCursor {
+ t.Helper()
+ // A pooled cursor, whose creator reference is dropped when the test ends, after
+ // the batches that pin it were released.
+ bc := generateBlockCursor()
+ t.Cleanup(func() { releaseBlockCursor(bc) })
+ bc.bm = blockMetadata{seriesID: common.SeriesID(7)}
+ bc.shardID = 3
+ bc.schemaTagTypes = map[string]pbv1.ValueType{"t_int": pbv1.ValueTypeInt64, "t_str": pbv1.ValueTypeStr}
+ for i := range typedTestRows {
+ bc.timestamps = append(bc.timestamps, int64(i+1))
+ bc.versions = append(bc.versions, int64(100+i))
+ }
+ load := func(name string, vt pbv1.ValueType, values [][]byte) column {
+ out := column{}
+ readBack(t, &column{name: name, valueType: vt, values: values}, &out, typedNumeric)
+ return out
+ }
+ strs := make([][]byte, typedTestRows)
+ for i := range strs {
+ strs[i] = []byte(fmt.Sprintf("s%d", i))
+ }
+ bc.tagFamilies = []columnFamily{{name: "default", columns: []column{
+ load("t_int", pbv1.ValueTypeInt64, genTypedValues(pbv1.ValueTypeInt64, kind, typedTestRows, withNull)),
+ load("t_str", pbv1.ValueTypeStr, strs),
+ }}}
+ bc.fields = columnFamily{columns: []column{
+ load("f_int", pbv1.ValueTypeInt64, genTypedValues(pbv1.ValueTypeInt64, kind, typedTestRows, withNull)),
+ load("f_float", pbv1.ValueTypeFloat64, genTypedValues(pbv1.ValueTypeFloat64, kind, typedTestRows, withNull)),
+ }}
+ return bc
+}
+
+func typedTestSchema(native bool) *vectorized.BatchSchema {
+ tagInt, tagStr, fInt, fFloat := vectorized.ColumnTypeTagValue, vectorized.ColumnTypeTagValue,
+ vectorized.ColumnTypeFieldValue, vectorized.ColumnTypeFieldValue
+ if native {
+ tagInt, tagStr, fInt, fFloat = vectorized.ColumnTypeInt64, vectorized.ColumnTypeString,
+ vectorized.ColumnTypeInt64, vectorized.ColumnTypeFloat64
+ }
+ return vectorized.NewBatchSchema([]vectorized.ColumnDef{
+ {Role: vectorized.RoleTimestamp, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleVersion, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleSeriesID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleShardID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_int", Type: tagInt},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_str", Type: tagStr},
+ {Role: vectorized.RoleField, Name: "f_int", Type: fInt},
+ {Role: vectorized.RoleField, Name: "f_float", Type: fFloat},
+ })
+}
+
+// requireBatchesEqual compares two batches cell by cell, including nulls.
+func requireBatchesEqual(t *testing.T, want, got *model.MeasureBatch) {
+ t.Helper()
+ require.Equal(t, want.Timestamps, got.Timestamps)
+ require.Equal(t, want.Versions, got.Versions)
+ require.Equal(t, want.SeriesIDs, got.SeriesIDs)
+ require.Equal(t, want.ShardIDs, got.ShardIDs)
+ cols := func(b *model.MeasureBatch) []vectorized.Column {
+ return append(append([]vectorized.Column{}, b.Tags...), b.Fields...)
+ }
+ wantCols, gotCols := cols(want), cols(got)
+ require.Len(t, gotCols, len(wantCols))
+ for c := range wantCols {
+ require.Equal(t, wantCols[c].Type(), gotCols[c].Type(), "column %d", c)
+ require.Equal(t, wantCols[c].Len(), gotCols[c].Len(), "column %d", c)
+ for i := 0; i < wantCols[c].Len(); i++ {
+ require.Equal(t, wantCols[c].IsNull(i), gotCols[c].IsNull(i), "column %d row %d null", c, i)
+ switch w := wantCols[c].(type) {
+ case *vectorized.TypedColumn[int64]:
+ require.Equal(t, w.Data()[i], gotCols[c].(*vectorized.TypedColumn[int64]).Data()[i], "column %d row %d", c, i)
+ case *vectorized.TypedColumn[float64]:
+ require.Equal(t, math.Float64bits(w.Data()[i]),
+ math.Float64bits(gotCols[c].(*vectorized.TypedColumn[float64]).Data()[i]), "column %d row %d", c, i)
+ case *vectorized.TypedColumn[string]:
+ require.Equal(t, w.Data()[i], gotCols[c].(*vectorized.TypedColumn[string]).Data()[i], "column %d row %d", c, i)
+ case *vectorized.TypedColumn[*modelv1.TagValue]:
+ require.True(t, proto.Equal(w.Data()[i], gotCols[c].(*vectorized.TypedColumn[*modelv1.TagValue]).Data()[i]),
+ "column %d row %d", c, i)
+ case *vectorized.TypedColumn[*modelv1.FieldValue]:
+ require.True(t, proto.Equal(w.Data()[i], gotCols[c].(*vectorized.TypedColumn[*modelv1.FieldValue]).Data()[i]),
+ "column %d row %d", c, i)
+ default:
+ t.Fatalf("unexpected column type %T", w)
+ }
+ }
+ }
+}
+
+func TestCopyAllToBatchTypedMatchesBytes(t *testing.T) {
+ verifyNoScanLeaks(t)
+ hidden := map[common.SeriesID]map[string]*modelv1.TagValue{
+ 7: {"t_int": {Value: &modelv1.TagValue_Int{Int: &modelv1.Int{Value: 4242}}}},
+ }
+ for _, native := range []bool{true, false} {
+ for _, kind := range []string{"counter", "gauge"} {
+ for _, withNull := range []bool{false, true} {
+ for _, desc := range []bool{false, true} {
+ for _, mode := range []string{"plain", "hidden", "schemaMismatch"} {
+ name := fmt.Sprintf("native=%v/%s/null=%v/desc=%v/%s", native, kind, withNull, desc, mode)
+ t.Run(name, func(t *testing.T) {
+ schema := typedTestSchema(native)
+ var index map[common.SeriesID]map[string]*modelv1.TagValue
+ build := func(typed bool) *blockCursor {
+ bc := buildTypedTestCursor(t, kind, withNull, typed)
+ bc.idx = typedTestRows / 4
+ if desc {
+ bc.idx = typedTestRows - typedTestRows/4
+ }
+ switch mode {
+ case "hidden":
+ index = hidden
+ case "schemaMismatch":
+ bc.schemaTagTypes["t_int"] = pbv1.ValueTypeStr
+ }
+ return bc
+ }
+ want := newMeasureBatchForSchema(schema, typedTestRows)
+ defer want.Release()
+ build(false).copyAllToBatch(want, schema, index, desc)
+ typedCursor := build(true)
+ if !withNull {
+ require.Equal(t, numericFormInt64, typedCursor.fields.columns[0].form)
+ require.Equal(t, numericFormFloat64, typedCursor.fields.columns[1].form)
+ }
+ got := newMeasureBatchForSchema(schema, typedTestRows)
+ defer got.Release()
+ typedCursor.copyAllToBatch(got, schema, index, desc)
+ require.Positive(t, got.RowCount())
+ requireBatchesEqual(t, want, got)
+ })
+ }
+ }
+ }
+ }
+ }
+}
+
+func TestCopyAndReplaceInBatchTypedMatchesBytes(t *testing.T) {
+ verifyNoScanLeaks(t)
+ hidden := map[common.SeriesID]map[string]*modelv1.TagValue{
+ 7: {"t_int": {Value: &modelv1.TagValue_Int{Int: &modelv1.Int{Value: 4242}}}},
+ }
+ for _, native := range []bool{true, false} {
+ for _, withNull := range []bool{false, true} {
+ for _, useHidden := range []bool{false, true} {
+ t.Run(fmt.Sprintf("native=%v/null=%v/hidden=%v", native, withNull, useHidden), func(t *testing.T) {
+ schema := typedTestSchema(native)
+ var index map[common.SeriesID]map[string]*modelv1.TagValue
+ if useHidden {
+ index = hidden
+ }
+ run := func(typed bool) *model.MeasureBatch {
+ bc := buildTypedTestCursor(t, "gauge", withNull, typed)
+ b := newMeasureBatchForSchema(schema, typedTestRows)
+ for _, row := range []int{0, 5, 6, typedTestRows / 2, typedTestRows - 1} {
+ bc.idx = row
+ bc.copyToBatch(b, schema, index)
+ }
+ // Overwrite the last row with other rows, as a newer duplicate version would.
+ for _, row := range []int{9, typedTestRows / 2} {
+ bc.idx = row
+ bc.replaceInBatch(b, schema, index)
+ }
+ return b
+ }
+ want, got := run(false), run(true)
+ defer want.Release()
+ defer got.Release()
+ requireBatchesEqual(t, want, got)
+ })
+ }
+ }
+ }
+}
+
+// TestLoadDataTypedMatchesBytes runs real parts through loadData in both modes
+// and checks the batches are identical, and that the typed mode really is typed.
+func TestLoadDataTypedMatchesBytes(t *testing.T) {
+ verifyNoScanLeaks(t)
+ h := setupBenchStorageHarness(t)
+ for name, schema := range map[string]*vectorized.BatchSchema{"native": benchNativeSchema(), "passthrough": benchPassthroughSchema()} {
+ for _, desc := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%s/desc=%v", name, desc), func(t *testing.T) {
+ collect := func(typed bool) []*model.MeasureBatch {
+ var out []*model.MeasureBatch
+ h.drainCursors(typed, func(bc *blockCursor) {
+ if typed {
+ require.Equal(t, numericFormInt64, bc.fields.columns[0].form)
+ require.Equal(t, numericFormInt64, bc.tagFamilies[0].columns[1].form)
+ } else {
+ require.Equal(t, numericFormBytes, bc.fields.columns[0].form)
+ }
+ if desc {
+ bc.idx = len(bc.timestamps) - 1
+ }
+ b := newMeasureBatchForSchema(schema, len(bc.timestamps))
+ bc.copyAllToBatch(b, schema, nil, desc)
+ out = append(out, b)
+ })
+ return out
+ }
+ want, got := collect(false), collect(true)
+ // A batch pins the cursor it borrowed from; release them so the cursors
+ // return to the pool.
+ defer func() {
+ for _, batches := range [][]*model.MeasureBatch{want, got} {
+ for _, b := range batches {
+ b.Release()
+ }
+ }
+ }()
+ require.NotEmpty(t, want)
+ require.Len(t, got, len(want))
+ for i := range want {
+ requireBatchesEqual(t, want[i], got[i])
+ }
+ })
+ }
+ }
+}
diff --git a/banyand/measure/const_verify_test.go b/banyand/measure/const_verify_test.go
new file mode 100644
index 000000000..3912c394b
--- /dev/null
+++ b/banyand/measure/const_verify_test.go
@@ -0,0 +1,42 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "flag"
+ "os"
+
+ vmeasure "github.com/apache/skywalking-banyandb/pkg/query/vectorized/measure"
+)
+
+// configureInvariantChecks runs every test of the package with the invariants of
+// ConstColumns and runs checked against every row: an entity tag carried as a constant
+// must equal the tag of every row it stands for, and every bucket run must hold only
+// rows of its bucket. The checks cost a pass over every batch, so a benchmark run, and
+// the bench gates, which time the scan, leave them off.
+func configureInvariantChecks() {
+ flag.Parse()
+ if bench := flag.Lookup("test.bench"); bench != nil && bench.Value.String() != "" {
+ return
+ }
+ if os.Getenv("RUN_BENCH_GATES") == "1" {
+ return
+ }
+ verifyEntityConstants.Store(true)
+ vmeasure.SetRunsVerification(true)
+}
diff --git a/banyand/measure/cursor_borrow_benchmark_test.go b/banyand/measure/cursor_borrow_benchmark_test.go
new file mode 100644
index 000000000..12c9c1129
--- /dev/null
+++ b/banyand/measure/cursor_borrow_benchmark_test.go
@@ -0,0 +1,86 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+const benchStrRows = 8192
+
+// benchStrCases are the 8,192-row string tag columns the borrowed fill is measured on.
+var benchStrCases = []struct {
+ gen func(i int) string
+ name string
+}{
+ // Short entity-style values, one distinct value per row: plain encoding.
+ {name: "short-plain", gen: func(i int) string { return fmt.Sprintf("svc-%06d", i) }},
+ // Short values with few distinct ones: dictionary encoding.
+ {name: "short-dict", gen: func(i int) string { return fmt.Sprintf("svc-%02d", i%40) }},
+ // Longer endpoint-style values (~43 bytes), unique per row: plain encoding.
+ {name: "endpoint-plain", gen: func(i int) string { return fmt.Sprintf("/api/v1/tenants/%05d/orders/%07d/items", i%900, i) }},
+}
+
+// BenchmarkStringTagFill measures decoding one encoded string tag column into a
+// cursor and filling a batch column from it, per 8,192-row block. "copy" is the
+// per-value string allocation fillTagRows does; "borrow" is borrowTagRows, which
+// builds string headers over the decoded bytes and pins the cursor.
+func BenchmarkStringTagFill(b *testing.B) {
+ schema := vectorized.NewBatchSchema([]vectorized.ColumnDef{
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t", Type: vectorized.ColumnTypeString},
+ })
+ def := schema.Columns[0]
+ for _, tc := range benchStrCases {
+ values := make([][]byte, benchStrRows)
+ for i := range values {
+ values[i] = []byte(tc.gen(i))
+ }
+ src := &column{name: "t", valueType: pbv1.ValueTypeStr, values: values}
+ cm := &columnMetadata{}
+ buf := &bytes.Buffer{}
+ w := &writer{}
+ w.init(buf)
+ src.mustWriteTo(cm, w)
+
+ for _, mode := range []string{"copy", "borrow"} {
+ b.Run(tc.name+"/"+mode, func(b *testing.B) {
+ bc := generateBlockCursor()
+ var col column
+ b.ReportAllocs()
+ b.ResetTimer()
+ for b.Loop() {
+ bc.columnValuesDecoder.Reset()
+ col.reset()
+ col.mustReadValues(&bc.columnValuesDecoder, buf, *cm, benchStrRows, true)
+ mb := newMeasureBatchForSchema(schema, benchStrRows)
+ if mode == "borrow" {
+ bc.borrowTagRows(mb, mb.Tags[0], def, &col, 0, benchStrRows, false)
+ } else {
+ fillTagRows(mb.Tags[0], def, &col, 0, benchStrRows, false)
+ }
+ mb.Release()
+ }
+ })
+ }
+ }
+}
diff --git a/banyand/measure/cursor_borrow_test.go b/banyand/measure/cursor_borrow_test.go
new file mode 100644
index 000000000..18c471da4
--- /dev/null
+++ b/banyand/measure/cursor_borrow_test.go
@@ -0,0 +1,292 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "strings"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+const borrowRows = 600
+
+// borrowFixture is a cursor whose tag columns went through a real encode/decode
+// round trip into the cursor's own decoder, so the decoded values alias the memory
+// a query cursor owns, plus what each row should read back as.
+type borrowFixture struct {
+ bc *blockCursor
+ uniq []string // plain-encoded string tag, one distinct value per row
+ dict []string // dictionary-encoded string tag; "" and null rows included
+ dictOK []bool // false where the dict row is null
+ blob [][]byte // plain-encoded bytes tag
+ arr [][]string // string-array tag; nil for an empty array
+}
+
+func (f *borrowFixture) schema() *vectorized.BatchSchema {
+ return vectorized.NewBatchSchema([]vectorized.ColumnDef{
+ {Role: vectorized.RoleTimestamp, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleVersion, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleSeriesID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleShardID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_uniq", Type: vectorized.ColumnTypeString},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_dict", Type: vectorized.ColumnTypeString},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_blob", Type: vectorized.ColumnTypeBytes},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_arr", Type: vectorized.ColumnTypeStrArray},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_int", Type: vectorized.ColumnTypeInt64},
+ })
+}
+
+func newBorrowFixture(t *testing.T) *borrowFixture {
+ t.Helper()
+ f := &borrowFixture{bc: generateBlockCursor()}
+ bc := f.bc
+ bc.bm = blockMetadata{seriesID: common.SeriesID(9)}
+ bc.shardID = 2
+ bc.schemaTagTypes = map[string]pbv1.ValueType{
+ "t_uniq": pbv1.ValueTypeStr, "t_dict": pbv1.ValueTypeStr, "t_blob": pbv1.ValueTypeBinaryData,
+ "t_arr": pbv1.ValueTypeStrArr, "t_int": pbv1.ValueTypeInt64,
+ }
+ rawUniq := make([][]byte, borrowRows)
+ rawDict := make([][]byte, borrowRows)
+ rawBlob := make([][]byte, borrowRows)
+ rawArr := make([][]byte, borrowRows)
+ rawInt := make([][]byte, borrowRows)
+ dictVals := []string{"alpha", "beta", "", "a-somewhat-longer-dictionary-value"}
+ for i := range borrowRows {
+ bc.timestamps = append(bc.timestamps, int64(i+1))
+ bc.versions = append(bc.versions, 1)
+ u := fmt.Sprintf("/api/v1/endpoint/%05d/%s", i, strings.Repeat("x", i%7))
+ f.uniq = append(f.uniq, u)
+ rawUniq[i] = []byte(u)
+
+ switch {
+ case i%11 == 0:
+ f.dict = append(f.dict, "")
+ f.dictOK = append(f.dictOK, false)
+ rawDict[i] = nil
+ default:
+ d := dictVals[i%len(dictVals)]
+ f.dict = append(f.dict, d)
+ f.dictOK = append(f.dictOK, true)
+ rawDict[i] = []byte(d)
+ }
+
+ blob := []byte(fmt.Sprintf("blob-%d-%s", i, strings.Repeat("\x01", i%5)))
+ f.blob = append(f.blob, blob)
+ rawBlob[i] = append([]byte(nil), blob...)
+
+ var elems []string
+ switch i % 4 {
+ case 0:
+ elems = []string{"one", fmt.Sprintf("two-%d", i)}
+ case 1:
+ elems = []string{"has|pipe", `back\slash`, "plain"}
+ case 2:
+ elems = nil
+ case 3:
+ elems = []string{""}
+ }
+ f.arr = append(f.arr, elems)
+ var enc []byte
+ for _, e := range elems {
+ enc = marshalVarArray(enc, []byte(e))
+ }
+ if elems == nil {
+ enc = []byte{}
+ }
+ rawArr[i] = enc
+ rawInt[i] = []byte{0, 0, 0, 0, 0, 0, 0, byte(i)}
+ }
+ load := func(name string, vt pbv1.ValueType, values [][]byte) column {
+ src := &column{name: name, valueType: vt, values: values}
+ cm := &columnMetadata{}
+ buf := &bytes.Buffer{}
+ w := &writer{}
+ w.init(buf)
+ src.mustWriteTo(cm, w)
+ dst := column{}
+ dst.mustReadValues(&bc.columnValuesDecoder, buf, *cm, uint64(len(values)), true)
+ return dst
+ }
+ bc.tagFamilies = []columnFamily{{name: "default", columns: []column{
+ load("t_uniq", pbv1.ValueTypeStr, rawUniq),
+ load("t_dict", pbv1.ValueTypeStr, rawDict),
+ load("t_blob", pbv1.ValueTypeBinaryData, rawBlob),
+ load("t_arr", pbv1.ValueTypeStrArr, rawArr),
+ load("t_int", pbv1.ValueTypeInt64, rawInt),
+ }}}
+ return f
+}
+
+func (f *borrowFixture) requireBatch(t *testing.T, b *model.MeasureBatch, desc bool) {
+ t.Helper()
+ uniq := b.Tags[0].(*vectorized.TypedColumn[string])
+ dict := b.Tags[1].(*vectorized.TypedColumn[string])
+ blob := b.Tags[2].(*vectorized.TypedColumn[[]byte])
+ arr := b.Tags[3].(*vectorized.TypedColumn[[]string])
+ require.Equal(t, borrowRows, uniq.Len())
+ for pos := range borrowRows {
+ src := pos
+ if desc {
+ src = borrowRows - 1 - pos
+ }
+ require.Equal(t, f.uniq[src], uniq.Data()[pos], "uniq row %d", pos)
+ require.Equal(t, !f.dictOK[src], dict.IsNull(pos), "dict null row %d", pos)
+ if f.dictOK[src] {
+ require.Equal(t, f.dict[src], dict.Data()[pos], "dict row %d", pos)
+ }
+ require.Equal(t, f.blob[src], blob.Data()[pos], "blob row %d", pos)
+ require.Equal(t, f.arr[src], arr.Data()[pos], "arr row %d", pos)
+ }
+}
+
+func TestBlockCursorRefcountLifecycle(t *testing.T) {
+ verifyNoScanLeaks(t)
+ bc := generateBlockCursor()
+ require.EqualValues(t, 1, bc.refs.Load())
+ bc.timestamps = append(bc.timestamps, 1, 2, 3)
+
+ bc.Retain()
+ require.EqualValues(t, 2, bc.refs.Load())
+ releaseBlockCursor(bc)
+ require.EqualValues(t, 1, bc.refs.Load())
+ require.Len(t, bc.timestamps, 3, "a cursor with a reference left must not be reset")
+
+ bc.Release()
+ require.EqualValues(t, 0, bc.refs.Load())
+ require.Empty(t, bc.timestamps, "the last release resets the cursor")
+
+ require.Panics(t, func() { releaseBlockCursor(bc) }, "a release past the last reference is detected")
+ require.EqualValues(t, 0, bc.refs.Load(), "a detected double release must not leave the count negative")
+}
+
+func TestBlockCursorPoisonsOnLastRelease(t *testing.T) {
+ verifyNoScanLeaks(t)
+ f := newBorrowFixture(t)
+ view := f.bc.tagFamilies[0].columns[0].values[3]
+ want := string(view)
+ require.NotEmpty(t, want)
+
+ f.bc.Retain()
+ releaseBlockCursor(f.bc)
+ require.Equal(t, want, string(view), "memory stays intact while a reference is held")
+
+ releaseBlockCursor(f.bc)
+ require.Equal(t, strings.Repeat("\xdd", len(want)), string(view),
+ "memory is overwritten with the sentinel when the last reference goes")
+}
+
+func TestBorrowTagRowsMatchesCopy(t *testing.T) {
+ verifyNoScanLeaks(t)
+ for _, desc := range []bool{false, true} {
+ t.Run(fmt.Sprintf("desc=%v", desc), func(t *testing.T) {
+ f := newBorrowFixture(t)
+ if desc {
+ f.bc.idx = borrowRows - 1 // loadCursorsForBatch positions a descending cursor at its last row.
+ }
+ schema := f.schema()
+ b := newMeasureBatchForSchema(schema, borrowRows)
+ f.bc.copyAllToBatch(b, schema, nil, desc)
+ f.requireBatch(t, b, desc)
+ require.Equal(t, 1, b.Owners.Len())
+ require.EqualValues(t, 2, f.bc.refs.Load(), "the batch pins the cursor once, however many columns borrow")
+
+ // Release the query's reference: the batch keeps the cursor alive.
+ releaseBlockCursor(f.bc)
+ require.EqualValues(t, 1, f.bc.refs.Load())
+ f.requireBatch(t, b, desc)
+
+ // Capture header copies, then release the batch: the cursor is recycled and
+ // poisoned, which a header kept past the release observes.
+ uniq := b.Tags[0].(*vectorized.TypedColumn[string])
+ dict := b.Tags[1].(*vectorized.TypedColumn[string])
+ const pos = 3
+ stale, staleDict := uniq.Data()[pos], dict.Data()[pos]
+ wantStale, wantStaleDict := f.uniq[pos], f.dict[pos]
+ if desc {
+ wantStale, wantStaleDict = f.uniq[borrowRows-1-pos], f.dict[borrowRows-1-pos]
+ }
+ require.NotEmpty(t, wantStale)
+ require.NotEmpty(t, wantStaleDict)
+ b.Release()
+ require.EqualValues(t, 0, f.bc.refs.Load())
+ require.NotEqual(t, wantStale, stale, "plain-encoded cell must be poisoned after the last release")
+ require.NotEqual(t, wantStaleDict, staleDict, "dictionary-encoded cell must be poisoned after the last release")
+ require.Equal(t, strings.Repeat("\xdd", len(wantStale)), stale)
+ require.Equal(t, strings.Repeat("\xdd", len(wantStaleDict)), staleDict)
+ })
+ }
+}
+
+func TestBorrowTagRowsSkippedForPassthroughAndNumeric(t *testing.T) {
+ verifyNoScanLeaks(t)
+ f := newBorrowFixture(t)
+ schema := vectorized.NewBatchSchema([]vectorized.ColumnDef{
+ {Role: vectorized.RoleTimestamp, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleVersion, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleSeriesID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleShardID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_uniq", Type: vectorized.ColumnTypeTagValue},
+ {Role: vectorized.RoleTag, TagFamily: "default", Name: "t_int", Type: vectorized.ColumnTypeInt64},
+ })
+ b := newMeasureBatchForSchema(schema, borrowRows)
+ f.bc.copyAllToBatch(b, schema, nil, false)
+ require.Zero(t, b.Owners.Len(), "passthrough and numeric columns copy, so nothing is pinned")
+ require.EqualValues(t, 1, f.bc.refs.Load())
+ b.Release()
+ releaseBlockCursor(f.bc)
+}
+
+func TestBorrowTagRowsMergePathPinsEachCursorOnce(t *testing.T) {
+ verifyNoScanLeaks(t)
+ a, c := newBorrowFixture(t), newBorrowFixture(t)
+ schema := a.schema()
+ b := newMeasureBatchForSchema(schema, 8)
+ for i := range 6 {
+ src := a
+ if i%2 == 1 {
+ src = c
+ }
+ src.bc.idx = i
+ src.bc.copyToBatch(b, schema, nil)
+ }
+ require.Equal(t, 2, b.Owners.Len())
+ require.EqualValues(t, 2, a.bc.refs.Load())
+ require.EqualValues(t, 2, c.bc.refs.Load())
+ releaseBlockCursor(a.bc)
+ releaseBlockCursor(c.bc)
+ for i := range 6 {
+ src := a
+ if i%2 == 1 {
+ src = c
+ }
+ require.Equal(t, src.uniq[i], b.Tags[0].(*vectorized.TypedColumn[string]).Data()[i])
+ require.Equal(t, src.arr[i], b.Tags[3].(*vectorized.TypedColumn[[]string]).Data()[i])
+ }
+ b.Release()
+ require.EqualValues(t, 0, a.bc.refs.Load())
+ require.EqualValues(t, 0, c.bc.refs.Load())
+}
diff --git a/banyand/measure/leak_check_selftest_test.go b/banyand/measure/leak_check_selftest_test.go
new file mode 100644
index 000000000..4d1cc61d3
--- /dev/null
+++ b/banyand/measure/leak_check_selftest_test.go
@@ -0,0 +1,94 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+)
+
+// recordingTB captures what verifyNoScanLeaks would report, so the check itself is
+// tested: a leak check that never fails would pass every suite.
+type recordingTB struct {
+ testing.TB
+ cleanups []func()
+ errs []string
+}
+
+func (r *recordingTB) Helper() {}
+func (r *recordingTB) Cleanup(f func()) { r.cleanups = append(r.cleanups, f) }
+func (r *recordingTB) Errorf(f string, a ...any) { r.errs = append(r.errs, fmt.Sprintf(f, a...)) }
+
+func (r *recordingTB) finish() {
+ for i := len(r.cleanups) - 1; i >= 0; i-- {
+ r.cleanups[i]()
+ }
+}
+
+func TestVerifyNoScanLeaksReportsEachKindOfLeak(t *testing.T) {
+ prev := scanLeakSettle
+ scanLeakSettle = 50 * time.Millisecond
+ defer func() { scanLeakSettle = prev }()
+
+ t.Run("clean", func(t *testing.T) {
+ r := &recordingTB{TB: t}
+ verifyNoScanLeaks(r)
+ bc := generateBlockCursor()
+ releaseBlockCursor(bc)
+ r.finish()
+ require.Empty(t, r.errs)
+ })
+ t.Run("cursor still referenced", func(t *testing.T) {
+ r := &recordingTB{TB: t}
+ verifyNoScanLeaks(r)
+ bc := generateBlockCursor()
+ r.finish()
+ releaseBlockCursor(bc)
+ require.Len(t, r.errs, 1)
+ })
+ t.Run("reservation still held", func(t *testing.T) {
+ r := &recordingTB{TB: t}
+ verifyNoScanLeaks(r)
+ budget := newScanBudget(-1, nil)
+ var res scanReservation
+ require.Equal(t, reserveOK, budget.tryReserve(100, false, &res))
+ r.finish()
+ res.release()
+ require.Len(t, r.errs, 1)
+ })
+ t.Run("decode task still queued", func(t *testing.T) {
+ r := &recordingTB{TB: t}
+ verifyNoScanLeaks(r)
+ var task decodeTask
+ task.init(t.Context(), nil, nil, false, nil, nil)
+ r.finish()
+ require.True(t, task.cancel(errScanClosed))
+ require.Len(t, r.errs, 1)
+ })
+ t.Run("decode worker still alive", func(t *testing.T) {
+ r := &recordingTB{TB: t}
+ verifyNoScanLeaks(r)
+ p := newDecodePool(1, nil)
+ r.finish()
+ p.close()
+ require.Len(t, r.errs, 1)
+ })
+}
diff --git a/banyand/measure/leak_check_test.go b/banyand/measure/leak_check_test.go
new file mode 100644
index 000000000..03e1c78a6
--- /dev/null
+++ b/banyand/measure/leak_check_test.go
@@ -0,0 +1,107 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "os"
+ "testing"
+ "time"
+
+ "github.com/apache/skywalking-banyandb/pkg/pool"
+)
+
+// Leak checking for the vectorized scan (measure scan path v2, step 4d).
+//
+// verifyNoScanLeaks compares what the scan holds at the end of a test with what it
+// held at the start: a block cursor still referenced (by a scan or by a batch that
+// borrows its buffers), a scan reservation still held, a decode task still queued
+// or running, a decode worker still alive, an unreleased query result. The
+// comparison is against the starting point, not zero, because the suite shares one
+// decode pool and a Ginkgo suite may keep a service running across tests; TestMain
+// checks the absolute zero once everything has finished.
+
+// scanLeakSettle is how long a test end waits for the scan's goroutines to settle
+// before an outstanding resource is reported: a canceled decode finishes on a worker.
+var scanLeakSettle = 5 * time.Second
+
+// verifyNoScanLeaks fails t when the scan still holds, at the end of t, anything it
+// did not hold when verifyNoScanLeaks was called. The check runs in a t.Cleanup, so
+// it sees the state after the test's own cleanups, which were registered later.
+func verifyNoScanLeaks(t testing.TB) {
+ t.Helper()
+ // Start the shared pool first so its workers are part of the baseline.
+ sharedFixturePool()
+ before := ScanResourceSnapshot()
+ t.Cleanup(func() {
+ t.Helper()
+ var now ScanResources
+ deadline := time.Now().Add(scanLeakSettle)
+ for {
+ now = ScanResourceSnapshot()
+ if now == before || time.Now().After(deadline) {
+ break
+ }
+ time.Sleep(5 * time.Millisecond)
+ }
+ if now != before {
+ t.Errorf("the scan leaked resources: before=%+v after=%+v%s", before, now, leakStacks())
+ }
+ })
+}
+
+// leakStacks lists the acquisition stacks of the cursors still checked out, when
+// stack tracking is on.
+func leakStacks() string {
+ stacks := pool.AllStacks()[cursorPoolName]
+ if len(stacks) == 0 {
+ return ""
+ }
+ out := "\nblock cursors still checked out were acquired at:"
+ for _, s := range stacks {
+ out += "\n" + s
+ }
+ return out
+}
+
+// TestMain fails the package when, after every test and suite has run, the scan
+// still holds a cursor, a reservation, a decode task or a query result. Decode
+// workers are not counted: the shared fixture pool lives for the whole run.
+func TestMain(m *testing.M) {
+ configureInvariantChecks()
+ if os.Getenv("BYDB_LEAK_STACKS") == "1" {
+ pool.EnableStackTracking(true)
+ }
+ code := m.Run()
+ if code == 0 {
+ deadline := time.Now().Add(scanLeakSettle)
+ var res ScanResources
+ for {
+ res = ScanResourceSnapshot()
+ if res.Idle() || time.Now().After(deadline) {
+ break
+ }
+ time.Sleep(10 * time.Millisecond)
+ }
+ if !res.Idle() {
+ fmt.Fprintf(os.Stderr, "measure scan leak at the end of the test binary: %+v%s\n", res, leakStacks())
+ code = 1
+ }
+ }
+ os.Exit(code)
+}
diff --git a/banyand/measure/measure.go b/banyand/measure/measure.go
index 8dc28579c..98ca9225d 100644
--- a/banyand/measure/measure.go
+++ b/banyand/measure/measure.go
@@ -21,9 +21,6 @@
package measure
import (
- "errors"
- "fmt"
- "strconv"
"sync/atomic"
"time"
@@ -86,6 +83,7 @@ type measure struct {
tsdb atomic.Value
c storage.Cache
pm protector.Memory
+ decodePool *decodePool
l *logger.Logger
schema *databasev1.Measure
schemaRepo *schemaRepo
@@ -128,6 +126,8 @@ func (m *measure) parseSpec() (err error) {
type measureSpec struct {
schema *databasev1.Measure
+ // decodePool is the node's decode pool; nil decodes scans inline.
+ decodePool *decodePool
}
type queryMetrics struct {
@@ -136,6 +136,29 @@ type queryMetrics struct {
resultPoints meter.Histogram
totalQueryResultStarted meter.Counter
totalQueryResultFinished meter.Counter
+ // Series-window scan (scan path v2): how many series windows a query scanned
+ // and how many of them, by block metadata, had blocks that overlap in time.
+ // The segment counters split the scan's segments into the zero-copy fast path
+ // (one block) and the row merge (two or more overlapping blocks), which is
+ // the number that says whether overlap is rare on a real cluster.
+ scanWindows meter.Counter
+ scanWindowsOverlapped meter.Counter
+ scanSegmentsFast meter.Counter
+ scanSegmentsOverlap meter.Counter
+ // Time-ordered merge (scan path v2 step 4c): the rows and cursors the merge
+ // handled and, per merge, the most cursors it held decoded at once, which is
+ // what bounds the scan's memory.
+ scanMergeRows meter.Counter
+ scanMergeCursors meter.Counter
+ scanMergePeakLive meter.Histogram
+ // Decode pool and reservations (scan path v2 step 4b): the most decodes a scan had
+ // requested ahead of the one it was consuming, how long a decode waited in the
+ // node's queue, and how often prefetch waited for room in the query's memory
+ // budget or the cursor the scan needed could not be reserved.
+ scanPrefetchDepth meter.Histogram
+ scanDecodeQueueWait meter.Histogram
+ scanReservationsWaiting meter.Counter
+ scanReservationsFailed meter.Counter
}
func newQueryMetrics(factory observability.Factory) *queryMetrics {
@@ -148,6 +171,18 @@ func newQueryMetrics(factory observability.Factory) *queryMetrics {
resultPoints: factory.NewHistogram("result_points", meter.DefBuckets),
totalQueryResultStarted: factory.NewCounter("total_query_result_started"),
totalQueryResultFinished: factory.NewCounter("total_query_result_finished"),
+ scanWindows: factory.NewCounter("total_scan_windows"),
+ scanWindowsOverlapped: factory.NewCounter("total_scan_windows_overlapped"),
+ scanSegmentsFast: factory.NewCounter("total_scan_segments_fast"),
+ scanSegmentsOverlap: factory.NewCounter("total_scan_segments_overlap"),
+ scanMergeRows: factory.NewCounter("total_scan_merge_rows"),
+ scanMergeCursors: factory.NewCounter("total_scan_merge_cursors"),
+ scanMergePeakLive: factory.NewHistogram("scan_merge_peak_live_cursors", meter.DefBuckets),
+ scanPrefetchDepth: factory.NewHistogram("scan_prefetch_depth", meter.Buckets{1, 2, 3, 4, 6, 8, 12, 16, 24, 32}),
+ scanDecodeQueueWait: factory.NewHistogram("scan_decode_queue_wait_seconds",
+ meter.Buckets{.0001, .0005, .001, .005, .01, .025, .05, .1, .25, .5, 1}),
+ scanReservationsWaiting: factory.NewCounter("total_scan_reservations_waiting"),
+ scanReservationsFailed: factory.NewCounter("total_scan_reservations_failed"),
}
}
@@ -160,6 +195,7 @@ func openMeasure(spec measureSpec,
l: l,
c: c,
pm: pm,
+ decodePool: spec.decodePool,
schemaRepo: schemaRepo,
queryMetrics: qm,
vectorized: vectorized,
@@ -177,35 +213,10 @@ func (m *measure) VectorizedConfig() vmeasure.VectorizedConfig {
return m.vectorized
}
-// removedRowQueryFlag accepts --<engine>-vectorized-enabled=true as a no-op and
-// rejects =false, so an operator whose rollback command line survived the
-// upgrade gets a message naming the issue instead of "unknown flag". The flag
-// is removed entirely in 0.13.0.
-type removedRowQueryFlag string
-
-func (f *removedRowQueryFlag) Set(value string) error {
- parsed, parseErr := strconv.ParseBool(value)
- if parseErr != nil {
- return fmt.Errorf("invalid boolean value %q: %w", value, parseErr)
- }
- if !parsed {
- return errors.New("row-based query was removed in 0.12.0, see apache/skywalking#13998")
- }
- *f = removedRowQueryFlag(value)
- return nil
-}
-
-func (f *removedRowQueryFlag) String() string { return string(*f) }
-
-func (*removedRowQueryFlag) Type() string { return "bool" }
-
// bindVectorizedFlags wires VectorizedConfig fields to a run.FlagSet. The
// defaults match vmeasure.DefaultConfig.
func bindVectorizedFlags(flagS *run.FlagSet, cfg *vmeasure.VectorizedConfig) {
defaults := vmeasure.DefaultConfig()
- removed := removedRowQueryFlag("true")
- flagS.Var(&removed, "measure-vectorized-enabled", "deprecated no-op; removed in 0.13.0")
- flagS.Lookup("measure-vectorized-enabled").NoOptDefVal = "true"
flagS.IntVar(&cfg.BatchSize, "measure-vectorized-batch-size", defaults.BatchSize,
"row count per vectorized batch")
flagS.IntVar(&cfg.QueryMemoryMiB, "measure-vectorized-query-memory-mib", defaults.QueryMemoryMiB,
diff --git a/banyand/measure/metadata.go b/banyand/measure/metadata.go
index fd873b5e3..a0ee0d3c8 100644
--- a/banyand/measure/metadata.go
+++ b/banyand/measure/metadata.go
@@ -24,7 +24,6 @@ import (
"path"
"strings"
"sync"
- "sync/atomic"
"time"
"github.com/cenkalti/backoff/v4"
@@ -700,17 +699,35 @@ func (sr *schemaRepo) stopAllProcessorsWithGroupPrefix(groupName string) {
var _ resourceSchema.ResourceSupplier = (*supplier)(nil)
type supplier struct {
- metadata metadata.Repo
- omr observability.MetricsRegistry
- c storage.Cache
- pm protector.Memory
- l *logger.Logger
- schemaRepo *schemaRepo
- nodeLabels map[string]string
- queryMetrics atomic.Pointer[queryMetrics]
- queryMetricsFactory atomic.Value
- path string
- option option
+ metadata metadata.Repo
+ omr observability.MetricsRegistry
+ c storage.Cache
+ pm protector.Memory
+ decodePool *decodePool
+ l *logger.Logger
+ schemaRepo *schemaRepo
+ nodeLabels map[string]string
+ // queryMetrics holds one set of query metrics per group, keyed by group name. A
+ // group's metrics carry its name as a label, so a node with several groups must
+ // keep every group's collectors registered, not only the last one opened.
+ queryMetrics sync.Map
+ path string
+ option option
+}
+
+// groupQueryMetrics is the query metrics of one group and the factory that owns
+// their collectors.
+type groupQueryMetrics struct {
+ metrics *queryMetrics
+ factory observability.Factory
+}
+
+// queryMetricsOf returns the query metrics of group, or nil before the group opens.
+func (s *supplier) queryMetricsOf(group string) *queryMetrics {
+ if v, ok := s.queryMetrics.Load(group); ok {
+ return v.(*groupQueryMetrics).metrics
+ }
+ return nil
}
func newSupplier(path string, svc *standalone, sr *schemaRepo, nodeLabels map[string]string) *supplier {
@@ -732,6 +749,7 @@ func newSupplier(path string, svc *standalone, sr *schemaRepo, nodeLabels map[st
option: opt,
omr: svc.omr,
pm: svc.pm,
+ decodePool: svc.decodePool,
schemaRepo: sr,
nodeLabels: nodeLabels,
}
@@ -740,8 +758,9 @@ func newSupplier(path string, svc *standalone, sr *schemaRepo, nodeLabels map[st
func (s *supplier) OpenResource(spec resourceSchema.Resource) (resourceSchema.IndexListener, error) {
measureSchema := spec.Schema().(*databasev1.Measure)
return openMeasure(measureSpec{
- schema: measureSchema,
- }, s.l, s.c, s.pm, s.schemaRepo, s.queryMetrics.Load(), s.option.vectorized)
+ schema: measureSchema,
+ decodePool: s.decodePool,
+ }, s.l, s.c, s.pm, s.schemaRepo, s.queryMetricsOf(measureSchema.GetMetadata().GetGroup()), s.option.vectorized)
}
func (s *supplier) ResourceSchema(md *commonv1.Metadata) (resourceSchema.ResourceSchema, error) {
diff --git a/banyand/measure/metrics.go b/banyand/measure/metrics.go
index 7c9fd508b..fdd000f0d 100644
--- a/banyand/measure/metrics.go
+++ b/banyand/measure/metrics.go
@@ -327,12 +327,13 @@ func (m *metrics) DeleteAll() {
func (s *supplier) newMetrics(p common.Position) (storage.Metrics, observability.Factory) {
factory := s.omr.With(measureScope.ConstLabels(meter.ToLabelPairs(common.DBLabelNames(), p.DBLabelValues())))
- if prev, ok := s.queryMetricsFactory.Load().(observability.Factory); ok && prev != nil {
- prev.Close()
+ // Reopening a group (for example after it was deleted and created again) replaces
+ // that group's collectors; other groups keep theirs.
+ if prev, ok := s.queryMetrics.Load(p.Database); ok {
+ prev.(*groupQueryMetrics).factory.Close()
}
queryFactory := s.omr.With(measureScope.SubScope("query").ConstLabels(meter.ToLabelPairs(common.DBLabelNames(), p.DBLabelValues())))
- s.queryMetricsFactory.Store(queryFactory)
- s.queryMetrics.Store(newQueryMetrics(queryFactory))
+ s.queryMetrics.Store(p.Database, &groupQueryMetrics{factory: queryFactory, metrics: newQueryMetrics(queryFactory)})
return &metrics{
totalWritten: factory.NewCounter("total_written"),
totalBatch: factory.NewCounter("total_batch"),
diff --git a/banyand/measure/migration_copy.go b/banyand/measure/migration_copy.go
index fd26fad14..93677b150 100644
--- a/banyand/measure/migration_copy.go
+++ b/banyand/measure/migration_copy.go
@@ -829,7 +829,7 @@ func slowCopyOnePart(in processPartInput, srcPartID uint64) (processPartResult,
bm := &bms[j]
bm.tagProjection = in.tagProjection
b := generateBlock()
- b.mustReadFrom(in.decoder, p, *bm, nil)
+ b.mustReadFrom(in.decoder, p, *bm, nil, false)
liveBlocks = append(liveBlocks, b)
for k := uint64(0); k < bm.count; k++ {
diff --git a/banyand/measure/native_projection_benchmark_test.go b/banyand/measure/native_projection_benchmark_test.go
new file mode 100644
index 000000000..23429350d
--- /dev/null
+++ b/banyand/measure/native_projection_benchmark_test.go
@@ -0,0 +1,355 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "fmt"
+ "os"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/banyand/internal/storage"
+ "github.com/apache/skywalking-banyandb/banyand/protector"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ "github.com/apache/skywalking-banyandb/pkg/fs"
+ "github.com/apache/skywalking-banyandb/pkg/logger"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+ vmeasure "github.com/apache/skywalking-banyandb/pkg/query/vectorized/measure"
+ "github.com/apache/skywalking-banyandb/pkg/test"
+ "github.com/apache/skywalking-banyandb/pkg/timestamp"
+)
+
+// End-to-end projection benchmark: real stored blocks are decoded by the
+// blockCursor straight into the batch the schema declares, drained through the
+// vectorized pipeline and serialized at egress. It compares the pre-native column
+// selection (every non-GroupBy/Agg projected column passthrough) against the
+// native default, so the decode savings and the egress wrapper cost are both
+// inside the measured loop.
+//
+// go test ./banyand/measure -run '^$' -bench BenchmarkNativeProjection -benchmem -count=6
+// BYDB_BENCH_LEGACY_SCHEMA=1 go test ... (the baseline)
+
+const (
+ nativeBenchSeries = 100
+ nativeBenchRowsPer = 1000
+ nativeBenchBatch = 1024
+ nativeBenchRows = nativeBenchSeries * nativeBenchRowsPer
+)
+
+var (
+ nativeBenchTagProj = []model.TagProjection{
+ {Family: "default", Names: []string{"svc", "endpoint", "env_id", "blob"}},
+ }
+ nativeBenchFieldProj = []string{"v_int", "v_float"}
+ nativeBenchTagTypes = map[string]pbv1.ValueType{
+ "svc": pbv1.ValueTypeStr, "endpoint": pbv1.ValueTypeStr,
+ "env_id": pbv1.ValueTypeInt64, "blob": pbv1.ValueTypeBinaryData,
+ }
+)
+
+func nativeBenchMeasure() *databasev1.Measure {
+ return &databasev1.Measure{
+ TagFamilies: []*databasev1.TagFamilySpec{{Name: "default", Tags: []*databasev1.TagSpec{
+ {Name: "svc", Type: databasev1.TagType_TAG_TYPE_STRING},
+ {Name: "endpoint", Type: databasev1.TagType_TAG_TYPE_STRING},
+ {Name: "env_id", Type: databasev1.TagType_TAG_TYPE_INT},
+ {Name: "blob", Type: databasev1.TagType_TAG_TYPE_DATA_BINARY},
+ }}},
+ Fields: []*databasev1.FieldSpec{
+ {Name: "v_int", FieldType: databasev1.FieldType_FIELD_TYPE_INT},
+ {Name: "v_float", FieldType: databasev1.FieldType_FIELD_TYPE_FLOAT},
+ },
+ }
+}
+
+func nativeBenchSchema(legacy bool, opts model.MeasureQueryOptions) *vectorized.BatchSchema {
+ schema, err := vmeasure.BuildBatchSchema(nativeBenchMeasure(), opts)
+ if err != nil {
+ panic(err)
+ }
+ if !legacy {
+ return schema
+ }
+ return passthroughSchema(schema, opts)
+}
+
+// passthroughSchema reproduces the column selection used before projected
+// tags and fields became native by default: only GroupBy keys and the Agg
+// target stay native, every other projected tag/field is a passthrough column.
+func passthroughSchema(schema *vectorized.BatchSchema, opts model.MeasureQueryOptions) *vectorized.BatchSchema {
+ cols := append([]vectorized.ColumnDef(nil), schema.Columns...)
+ for i := range cols {
+ switch cols[i].Role {
+ case vectorized.RoleTag:
+ isKey := opts.GroupBy != nil && opts.GroupBy.TagFamily == cols[i].TagFamily && containsString(opts.GroupBy.TagNames, cols[i].Name)
+ if !isKey {
+ cols[i].Type = vectorized.ColumnTypeTagValue
+ }
+ case vectorized.RoleField:
+ if opts.Agg == nil || opts.Agg.FieldName != cols[i].Name {
+ cols[i].Type = vectorized.ColumnTypeFieldValue
+ }
+ case vectorized.RoleTimestamp, vectorized.RoleVersion, vectorized.RoleSeriesID, vectorized.RoleShardID,
+ vectorized.RoleElementID, vectorized.RoleOrderKey:
+ }
+ }
+ return vectorized.NewBatchSchema(cols)
+}
+
+func containsString(list []string, s string) bool {
+ for _, e := range list {
+ if e == s {
+ return true
+ }
+ }
+ return false
+}
+
+type nativeBenchHarness struct {
+ tst *tsTable
+ snap *snapshot
+ parts []partRef
+ sids []common.SeriesID
+}
+
+func setupNativeBenchHarness(b testing.TB) *nativeBenchHarness {
+ b.Helper()
+ tmpPath, defFn := test.Space(require.New(b))
+ tst, err := newTSTable(
+ fs.NewLocalFileSystem(),
+ tmpPath, common.Position{},
+ logger.GetLogger("bench"),
+ timestamp.TimeRange{},
+ option{flushTimeout: 0, mergePolicy: newDefaultMergePolicyForTesting(), protector: protector.Nop{}},
+ nil,
+ )
+ require.NoError(b, err)
+ total := nativeBenchRows
+ points := &dataPoints{
+ seriesIDs: make([]common.SeriesID, total),
+ timestamps: make([]int64, total),
+ versions: make([]int64, total),
+ tagFamilies: make([][]nameValues, total),
+ fields: make([]nameValues, total),
+ }
+ for s := range nativeBenchSeries {
+ for r := range nativeBenchRowsPer {
+ i := s*nativeBenchRowsPer + r
+ points.seriesIDs[i] = common.SeriesID(s + 1)
+ points.timestamps[i] = int64(r + 1)
+ points.versions[i] = 1
+ points.tagFamilies[i] = []nameValues{{name: "default", values: []*nameValue{
+ {name: "svc", valueType: pbv1.ValueTypeStr, value: []byte(fmt.Sprintf("service-%d", s%8))},
+ {name: "endpoint", valueType: pbv1.ValueTypeStr, value: []byte(fmt.Sprintf("/api/v1/resource/%d/item/%d", s, r))},
+ {name: "env_id", valueType: pbv1.ValueTypeInt64, value: convert.Int64ToBytes(int64(s + 1))},
+ {name: "blob", valueType: pbv1.ValueTypeBinaryData, value: []byte{byte(r), byte(s), 1, 2, 3, 4}},
+ }}}
+ points.fields[i] = nameValues{name: "skipped", values: []*nameValue{
+ {name: "v_int", valueType: pbv1.ValueTypeInt64, value: convert.Int64ToBytes(int64(r*7 + s))},
+ {name: "v_float", valueType: pbv1.ValueTypeFloat64, value: convert.Float64ToBytes(float64(r) * 0.25)},
+ }}
+ }
+ }
+ tst.mustAddDataPoints(points)
+ time.Sleep(300 * time.Millisecond)
+ snap := tst.currentSnapshot()
+ require.NotNil(b, snap)
+ shardCache := storage.NewShardCache("bench-group", 0, 0)
+ parts, _ := snap.getParts(nil, shardCache, 0, 0, int64(nativeBenchRowsPer)+1)
+ require.NotEmpty(b, parts)
+ sids := make([]common.SeriesID, nativeBenchSeries)
+ for i := range sids {
+ sids[i] = common.SeriesID(i + 1)
+ }
+ b.Cleanup(func() {
+ snap.decRef()
+ tst.Close()
+ defFn()
+ })
+ return &nativeBenchHarness{tst: tst, snap: snap, parts: parts, sids: sids}
+}
+
+// cursorBatchResult serves one MeasureBatch per stored block through PullBatch,
+// decoding with the same copyAllToBatch the production queryResult uses.
+type cursorBatchResult struct {
+ schema *vectorized.BatchSchema
+ ti *tstIter
+ opts queryOptions
+}
+
+func (h *nativeBenchHarness) newResult(schema *vectorized.BatchSchema, opts model.MeasureQueryOptions) *cursorBatchResult {
+ qo := queryOptions{
+ schemaTagTypes: nativeBenchTagTypes,
+ minTimestamp: 0,
+ maxTimestamp: int64(nativeBenchRowsPer) + 1,
+ }
+ qo.MeasureQueryOptions.TagProjection = opts.TagProjection
+ qo.MeasureQueryOptions.FieldProjection = opts.FieldProjection
+ ti := &tstIter{}
+ ti.init(h.parts, h.sids, qo.minTimestamp, qo.maxTimestamp)
+ return &cursorBatchResult{schema: schema, ti: ti, opts: qo}
+}
+
+func (r *cursorBatchResult) PullBatch(context.Context) (*model.MeasureBatch, error) {
+ for r.ti.nextBlock() {
+ bc := generateBlockCursor()
+ p := r.ti.piHeap[0]
+ bc.init(p.p, p.curBlock, r.opts)
+ tmpBlock := generateBlock()
+ loaded := bc.loadData(tmpBlock, true)
+ releaseBlock(tmpBlock)
+ if !loaded {
+ releaseBlockCursor(bc)
+ continue
+ }
+ mb := newMeasureBatchForSchema(r.schema, len(bc.timestamps))
+ bc.copyAllToBatch(mb, r.schema, nil, false)
+ releaseBlockCursor(bc)
+ return mb, nil
+ }
+ return nil, nil
+}
+
+func (r *cursorBatchResult) Release() {}
+
+// nativeBenchStages is what a scenario adds on top of the scan source.
+type nativeBenchStages struct {
+ out *vectorized.BatchSchema
+ breakers []vectorized.BreakerOperator
+ fusible []vectorized.FusibleOperator
+}
+
+type nativeBenchBuild func(schema *vectorized.BatchSchema, tracker *vectorized.MemoryTracker) nativeBenchStages
+
+type nativeBenchScenario struct {
+ build nativeBenchBuild
+ name string
+ opts model.MeasureQueryOptions
+ frame bool
+}
+
+func groupByStages(opts model.MeasureQueryOptions) nativeBenchBuild {
+ return func(s *vectorized.BatchSchema, tracker *vectorized.MemoryTracker) nativeBenchStages {
+ ops, err := vmeasure.BuildOperators(opts, s, tracker, nativeBenchBatch, vmeasure.AggModeAll)
+ if err != nil {
+ panic(err)
+ }
+ return nativeBenchStages{out: ops[0].OutputSchema(), breakers: []vectorized.BreakerOperator{ops[0]}}
+ }
+}
+
+func nativeBenchScenarios() []nativeBenchScenario {
+ plain := model.MeasureQueryOptions{TagProjection: nativeBenchTagProj, FieldProjection: nativeBenchFieldProj}
+ limit := func(n uint32) nativeBenchBuild {
+ return func(s *vectorized.BatchSchema, _ *vectorized.MemoryTracker) nativeBenchStages {
+ return nativeBenchStages{out: s, fusible: []vectorized.FusibleOperator{vmeasure.NewBatchLimit(s, 0, n)}}
+ }
+ }
+ top := func(n int) nativeBenchBuild {
+ return func(s *vectorized.BatchSchema, _ *vectorized.MemoryTracker) nativeBenchStages {
+ idx, _ := s.FieldIndex("v_int")
+ return nativeBenchStages{out: s, breakers: []vectorized.BreakerOperator{vmeasure.NewBatchTop(s, idx, n, false, nativeBenchBatch)}}
+ }
+ }
+ none := func(s *vectorized.BatchSchema, _ *vectorized.MemoryTracker) nativeBenchStages {
+ return nativeBenchStages{out: s}
+ }
+ groupBy := model.MeasureQueryOptions{
+ TagProjection: []model.TagProjection{{Family: "default", Names: []string{"svc"}}},
+ FieldProjection: []string{"v_int"},
+ GroupBy: &model.MeasureGroupBy{TagFamily: "default", TagNames: []string{"svc"}},
+ Agg: &model.MeasureAgg{FieldName: "v_int", Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_SUM},
+ }
+ return []nativeBenchScenario{
+ {name: "scan100", opts: plain, build: none},
+ {name: "limit10", opts: plain, build: limit(nativeBenchRows / 10)},
+ {name: "limit1", opts: plain, build: limit(nativeBenchRows / 100)},
+ {name: "top10", opts: plain, build: top(nativeBenchRows / 10)},
+ {name: "top1", opts: plain, build: top(nativeBenchRows / 100)},
+ {name: "groupby", opts: groupBy, build: groupByStages(groupBy)},
+ {name: "frame_scan100", opts: plain, build: none, frame: true},
+ {name: "frame_top10", opts: plain, build: top(nativeBenchRows / 10), frame: true},
+ }
+}
+
+func runNativeBenchScenario(b *testing.B, h *nativeBenchHarness, sc nativeBenchScenario, legacy bool) {
+ b.Helper()
+ b.ReportAllocs()
+ b.ResetTimer()
+ var sink int
+ for b.Loop() {
+ schema := nativeBenchSchema(legacy, sc.opts)
+ res := h.newResult(schema, sc.opts)
+ pool := vectorized.NewBatchPool(schema, nativeBenchBatch)
+ source := vmeasure.NewBatchSourceFromBatchResult(res, schema, pool, nativeBenchBatch)
+ tracker := vectorized.NewMemoryTracker(256 * 1024 * 1024)
+ builder := vectorized.NewPipelineBuilder().WithMemoryTracker(tracker).From(source)
+ stages := sc.build(schema, tracker)
+ outSchema := stages.out
+ for _, br := range stages.breakers {
+ builder.Break(br)
+ }
+ for _, f := range stages.fusible {
+ builder.Apply(f)
+ }
+ pipeline, err := builder.Build()
+ if err != nil {
+ b.Fatal(err)
+ }
+ if err = pipeline.Init(context.Background()); err != nil {
+ b.Fatal(err)
+ }
+ if sc.frame {
+ body, frameErr := vmeasure.DrainPipelineToFrame(context.Background(), pipeline, outSchema)
+ if frameErr != nil {
+ b.Fatal(frameErr)
+ }
+ sink += len(body)
+ _ = pipeline.Close()
+ continue
+ }
+ it := vmeasure.NewIteratorFromPipeline(context.Background(), pipeline, outSchema, vectorized.NewBatchPool(outSchema, nativeBenchBatch))
+ for it.Next() {
+ sink += len(it.Current())
+ }
+ if err = it.Close(); err != nil {
+ b.Fatal(err)
+ }
+ }
+ nativeBenchSink = sink
+}
+
+var nativeBenchSink int
+
+// BenchmarkNativeProjection runs every scenario once with the column selection
+// chosen by BYDB_BENCH_LEGACY_SCHEMA (unset: native default; 1: legacy).
+func BenchmarkNativeProjection(b *testing.B) {
+ legacy := os.Getenv("BYDB_BENCH_LEGACY_SCHEMA") == "1"
+ h := setupNativeBenchHarness(b)
+ for _, sc := range nativeBenchScenarios() {
+ b.Run(sc.name, func(b *testing.B) { runNativeBenchScenario(b, h, sc, legacy) })
+ }
+}
diff --git a/banyand/measure/native_projection_test.go b/banyand/measure/native_projection_test.go
new file mode 100644
index 000000000..7e0b17879
--- /dev/null
+++ b/banyand/measure/native_projection_test.go
@@ -0,0 +1,344 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "fmt"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+const nativeProjRows = 48
+
+// nativeProjTag / nativeProjField describe one stored column of the fixture.
+type nativeProjTag struct {
+ name string
+ vt pbv1.ValueType
+ typ vectorized.ColumnType
+}
+
+var nativeProjTags = []nativeProjTag{
+ {"t_int", pbv1.ValueTypeInt64, vectorized.ColumnTypeInt64},
+ {"t_str", pbv1.ValueTypeStr, vectorized.ColumnTypeString},
+ {"t_bin", pbv1.ValueTypeBinaryData, vectorized.ColumnTypeBytes},
+ {"t_iarr", pbv1.ValueTypeInt64Arr, vectorized.ColumnTypeInt64Array},
+ {"t_sarr", pbv1.ValueTypeStrArr, vectorized.ColumnTypeStrArray},
+}
+
+var nativeProjFields = []nativeProjTag{
+ {"f_int", pbv1.ValueTypeInt64, vectorized.ColumnTypeInt64},
+ {"f_float", pbv1.ValueTypeFloat64, vectorized.ColumnTypeFloat64},
+ {"f_str", pbv1.ValueTypeStr, vectorized.ColumnTypeString},
+ {"f_bin", pbv1.ValueTypeBinaryData, vectorized.ColumnTypeBytes},
+}
+
+// nativeProjRaw returns the stored bytes of one fixture cell. Every column has
+// null rows (i%7==3) and the string / binary columns also have empty rows.
+func nativeProjRaw(vt pbv1.ValueType, i int) []byte {
+ if i%7 == 3 {
+ return nil
+ }
+ switch vt {
+ case pbv1.ValueTypeInt64:
+ return convert.Int64ToBytes(int64(i*3 - 20))
+ case pbv1.ValueTypeFloat64:
+ return convert.Float64ToBytes(float64(i) * 1.5)
+ case pbv1.ValueTypeStr:
+ if i%5 == 0 {
+ return []byte{}
+ }
+ return []byte(fmt.Sprintf("value-%d", i))
+ case pbv1.ValueTypeBinaryData:
+ if i%5 == 0 {
+ return []byte{}
+ }
+ return []byte{byte(i), 0, 0xff}
+ case pbv1.ValueTypeInt64Arr:
+ if i%4 == 0 {
+ return []byte{}
+ }
+ var out []byte
+ for k := range i % 3 {
+ out = append(out, convert.Int64ToBytes(int64(i+k))...)
+ }
+ return append(out, convert.Int64ToBytes(int64(-i))...)
+ case pbv1.ValueTypeStrArr:
+ var out []byte
+ switch i % 4 {
+ case 0:
+ return []byte{}
+ case 1:
+ out = marshalVarArray(out, []byte("a|b"))
+ default:
+ out = marshalVarArray(out, []byte("x"))
+ out = marshalVarArray(out, []byte(fmt.Sprintf("y-%d", i)))
+ }
+ return out
+ default:
+ }
+ return nil
+}
+
+func buildNativeProjCursor(t *testing.T, typedNumeric bool) *blockCursor {
+ t.Helper()
+ // The cursor comes from the pool and its creator reference is dropped when the
+ // test ends, after the batches that pin it were released: releasing a cursor the
+ // pool never handed out would unbalance the pool's count.
+ bc := generateBlockCursor()
+ t.Cleanup(func() { releaseBlockCursor(bc) })
+ bc.bm = blockMetadata{seriesID: common.SeriesID(5)}
+ bc.shardID = 1
+ bc.schemaTagTypes = map[string]pbv1.ValueType{}
+ for i := range nativeProjRows {
+ bc.timestamps = append(bc.timestamps, int64(i+1))
+ bc.versions = append(bc.versions, 1)
+ }
+ load := func(name string, vt pbv1.ValueType) column {
+ values := make([][]byte, nativeProjRows)
+ for i := range values {
+ values[i] = nativeProjRaw(vt, i)
+ }
+ src := &column{name: name, valueType: vt, values: values}
+ cm := &columnMetadata{}
+ buf := &bytes.Buffer{}
+ w := &writer{}
+ w.init(buf)
+ src.mustWriteTo(cm, w)
+ dst := column{}
+ dst.mustReadValues(&bc.columnValuesDecoder, buf, *cm, uint64(nativeProjRows), typedNumeric)
+ return dst
+ }
+ var tagCols []column
+ for _, tg := range nativeProjTags {
+ tagCols = append(tagCols, load(tg.name, tg.vt))
+ bc.schemaTagTypes[tg.name] = tg.vt
+ }
+ bc.tagFamilies = []columnFamily{{name: "default", columns: tagCols}}
+ for _, f := range nativeProjFields {
+ bc.fields.columns = append(bc.fields.columns, load(f.name, f.vt))
+ }
+ return bc
+}
+
+// nativeProjSchema declares every fixture column, native or passthrough, plus a
+// tag the cursor does not store and a family it does not have.
+func nativeProjSchema(native bool) *vectorized.BatchSchema {
+ cols := []vectorized.ColumnDef{
+ {Role: vectorized.RoleTimestamp, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleVersion, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleSeriesID, Type: vectorized.ColumnTypeInt64},
+ {Role: vectorized.RoleShardID, Type: vectorized.ColumnTypeInt64},
+ }
+ for _, tg := range nativeProjTags {
+ typ := vectorized.ColumnTypeTagValue
+ if native {
+ typ = tg.typ
+ }
+ cols = append(cols, vectorized.ColumnDef{Role: vectorized.RoleTag, TagFamily: "default", Name: tg.name, Type: typ})
+ }
+ nativeStr, nativeFieldStr := vectorized.ColumnTypeTagValue, vectorized.ColumnTypeFieldValue
+ if native {
+ nativeStr, nativeFieldStr = vectorized.ColumnTypeString, vectorized.ColumnTypeString
+ }
+ cols = append(cols,
+ vectorized.ColumnDef{Role: vectorized.RoleTag, TagFamily: "default", Name: "t_missing", Type: nativeStr},
+ vectorized.ColumnDef{Role: vectorized.RoleTag, TagFamily: "other", Name: "t_other", Type: nativeStr},
+ )
+ for _, f := range nativeProjFields {
+ typ := vectorized.ColumnTypeFieldValue
+ if native {
+ typ = f.typ
+ }
+ cols = append(cols, vectorized.ColumnDef{Role: vectorized.RoleField, Name: f.name, Type: typ})
+ }
+ cols = append(cols, vectorized.ColumnDef{Role: vectorized.RoleField, Name: "f_missing", Type: nativeFieldStr})
+ return vectorized.NewBatchSchema(cols)
+}
+
+// nativeCellAsTagValue is what the egress serializer produces for a cell of
+// either column shape. Native null cells become pbv1.NullTagValue, as at egress.
+func nativeCellAsTagValue(col vectorized.Column, row int) *modelv1.TagValue {
+ if pc, ok := col.(*vectorized.TypedColumn[*modelv1.TagValue]); ok {
+ if v := pc.Data()[row]; v != nil {
+ return v
+ }
+ return pbv1.NullTagValue
+ }
+ if col.IsNull(row) {
+ return pbv1.NullTagValue
+ }
+ switch c := col.(type) {
+ case *vectorized.TypedColumn[int64]:
+ return &modelv1.TagValue{Value: &modelv1.TagValue_Int{Int: &modelv1.Int{Value: c.Data()[row]}}}
+ case *vectorized.TypedColumn[string]:
+ return &modelv1.TagValue{Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: c.Data()[row]}}}
+ case *vectorized.TypedColumn[[]byte]:
+ return &modelv1.TagValue{Value: &modelv1.TagValue_BinaryData{BinaryData: c.Data()[row]}}
+ case *vectorized.TypedColumn[[]int64]:
+ return &modelv1.TagValue{Value: &modelv1.TagValue_IntArray{IntArray: &modelv1.IntArray{Value: c.Data()[row]}}}
+ case *vectorized.TypedColumn[[]string]:
+ return &modelv1.TagValue{Value: &modelv1.TagValue_StrArray{StrArray: &modelv1.StrArray{Value: c.Data()[row]}}}
+ }
+ panic(fmt.Sprintf("unexpected tag column %T", col))
+}
+
+func nativeCellAsFieldValue(col vectorized.Column, row int) *modelv1.FieldValue {
+ if pc, ok := col.(*vectorized.TypedColumn[*modelv1.FieldValue]); ok {
+ if v := pc.Data()[row]; v != nil {
+ return v
+ }
+ return pbv1.NullFieldValue
+ }
+ if col.IsNull(row) {
+ return pbv1.NullFieldValue
+ }
+ switch c := col.(type) {
+ case *vectorized.TypedColumn[int64]:
+ return &modelv1.FieldValue{Value: &modelv1.FieldValue_Int{Int: &modelv1.Int{Value: c.Data()[row]}}}
+ case *vectorized.TypedColumn[float64]:
+ return &modelv1.FieldValue{Value: &modelv1.FieldValue_Float{Float: &modelv1.Float{Value: c.Data()[row]}}}
+ case *vectorized.TypedColumn[string]:
+ return &modelv1.FieldValue{Value: &modelv1.FieldValue_Str{Str: &modelv1.Str{Value: c.Data()[row]}}}
+ case *vectorized.TypedColumn[[]byte]:
+ return &modelv1.FieldValue{Value: &modelv1.FieldValue_BinaryData{BinaryData: c.Data()[row]}}
+ }
+ panic(fmt.Sprintf("unexpected field column %T", col))
+}
+
+// requireNativeMatchesPassthrough checks every cell of the native batch against
+// the passthrough batch after egress-style materialization. Marshaled bytes are
+// compared as well so empty-versus-nil payloads cannot hide a wire difference.
+func requireNativeMatchesPassthrough(t *testing.T, want, got *model.MeasureBatch) {
+ t.Helper()
+ require.Equal(t, want.Timestamps, got.Timestamps)
+ require.Equal(t, want.Versions, got.Versions)
+ require.Equal(t, want.SeriesIDs, got.SeriesIDs)
+ require.Equal(t, want.ShardIDs, got.ShardIDs)
+ require.Len(t, got.Tags, len(want.Tags))
+ require.Len(t, got.Fields, len(want.Fields))
+ for c := range want.Tags {
+ require.Equal(t, want.Tags[c].Len(), got.Tags[c].Len(), "tag column %d", c)
+ for i := 0; i < want.Tags[c].Len(); i++ {
+ w, g := nativeCellAsTagValue(want.Tags[c], i), nativeCellAsTagValue(got.Tags[c], i)
+ require.True(t, proto.Equal(w, g), "tag column %d row %d: want %v got %v", c, i, w, g)
+ wb, _ := proto.Marshal(w)
+ gb, _ := proto.Marshal(g)
+ require.Equal(t, wb, gb, "tag column %d row %d wire", c, i)
+ }
+ }
+ for c := range want.Fields {
+ require.Equal(t, want.Fields[c].Len(), got.Fields[c].Len(), "field column %d", c)
+ for i := 0; i < want.Fields[c].Len(); i++ {
+ w, g := nativeCellAsFieldValue(want.Fields[c], i), nativeCellAsFieldValue(got.Fields[c], i)
+ require.True(t, proto.Equal(w, g), "field column %d row %d: want %v got %v", c, i, w, g)
+ wb, _ := proto.Marshal(w)
+ gb, _ := proto.Marshal(g)
+ require.Equal(t, wb, gb, "field column %d row %d wire", c, i)
+ }
+ }
+}
+
+// TestNativeProjectionMatchesPassthrough decodes the same stored cells into a
+// passthrough schema and the all-native schema BuildBatchSchema now emits, over
+// every tag/field type, null and empty cells, absent columns, hidden-tag
+// substitution, a schema/stored type mismatch, both directions and the
+// single-row / replace paths.
+func TestNativeProjectionMatchesPassthrough(t *testing.T) {
+ verifyNoScanLeaks(t)
+ hiddenIndex := map[common.SeriesID]map[string]*modelv1.TagValue{
+ 5: {
+ "t_str": {Value: &modelv1.TagValue_Str{Str: &modelv1.Str{Value: "from-index"}}},
+ "t_iarr": {Value: &modelv1.TagValue_IntArray{IntArray: &modelv1.IntArray{Value: []int64{1, 2}}}},
+ },
+ }
+ for _, typedNumeric := range []bool{false, true} {
+ for _, desc := range []bool{false, true} {
+ for _, mode := range []string{"plain", "hidden", "schemaMismatch"} {
+ t.Run(fmt.Sprintf("typed=%v/desc=%v/%s", typedNumeric, desc, mode), func(t *testing.T) {
+ var index map[common.SeriesID]map[string]*modelv1.TagValue
+ build := func() *blockCursor {
+ bc := buildNativeProjCursor(t, typedNumeric)
+ bc.idx = 3
+ switch mode {
+ case "hidden":
+ index = hiddenIndex
+ case "schemaMismatch":
+ bc.schemaTagTypes["t_sarr"] = pbv1.ValueTypeStr
+ bc.schemaTagTypes["t_int"] = pbv1.ValueTypeBinaryData
+ }
+ return bc
+ }
+ passSchema, natSchema := nativeProjSchema(false), nativeProjSchema(true)
+ want := newMeasureBatchForSchema(passSchema, nativeProjRows)
+ defer want.Release()
+ build().copyAllToBatch(want, passSchema, index, desc)
+ got := newMeasureBatchForSchema(natSchema, nativeProjRows)
+ defer got.Release()
+ nativeCursor := build()
+ nativeCursor.copyAllToBatch(got, natSchema, index, desc)
+ require.Positive(t, got.RowCount())
+ requireNativeMatchesPassthrough(t, want, got)
+ })
+ }
+ }
+ }
+}
+
+func TestNativeProjectionSingleRowAndReplaceMatchPassthrough(t *testing.T) {
+ verifyNoScanLeaks(t)
+ hiddenIndex := map[common.SeriesID]map[string]*modelv1.TagValue{
+ 5: {"t_sarr": {Value: &modelv1.TagValue_StrArray{StrArray: &modelv1.StrArray{Value: []string{"h"}}}}},
+ }
+ for _, typedNumeric := range []bool{false, true} {
+ for _, useHidden := range []bool{false, true} {
+ t.Run(fmt.Sprintf("typed=%v/hidden=%v", typedNumeric, useHidden), func(t *testing.T) {
+ var index map[common.SeriesID]map[string]*modelv1.TagValue
+ if useHidden {
+ index = hiddenIndex
+ }
+ run := func(schema *vectorized.BatchSchema) *model.MeasureBatch {
+ bc := buildNativeProjCursor(t, typedNumeric)
+ b := newMeasureBatchForSchema(schema, nativeProjRows)
+ for _, row := range []int{0, 3, 4, 10, nativeProjRows - 1} {
+ bc.idx = row
+ bc.copyToBatch(b, schema, index)
+ }
+ for _, row := range []int{3, 17} {
+ bc.idx = row
+ bc.replaceInBatch(b, schema, index)
+ }
+ return b
+ }
+ want, got := run(nativeProjSchema(false)), run(nativeProjSchema(true))
+ requireNativeMatchesPassthrough(t, want, got)
+ want.Release()
+ got.Release()
+ })
+ }
+ }
+}
diff --git a/banyand/measure/part.go b/banyand/measure/part.go
index 75cec6aed..9bdb0a0fd 100644
--- a/banyand/measure/part.go
+++ b/banyand/measure/part.go
@@ -64,11 +64,9 @@ type part struct {
tagFamilies map[string]fs.Reader
tagType tagType
seriesMetadata fs.Reader // Optional: series metadata reader
- cache storage.Cache
path string
primaryBlockMetadata []primaryBlockMetadata
partMetadata partMetadata
- shardID common.ShardID
}
func (p *part) close() {
diff --git a/banyand/measure/part_iter.go b/banyand/measure/part_iter.go
index 2985501fa..71eca1c88 100644
--- a/banyand/measure/part_iter.go
+++ b/banyand/measure/part_iter.go
@@ -46,12 +46,14 @@ type partIter struct {
sidIdx int
minTimestamp int64
maxTimestamp int64
+ shardID common.ShardID
}
func (pi *partIter) reset() {
pi.curBlock = nil
pi.p = nil
pi.c = nil
+ pi.shardID = 0
pi.sids = nil
pi.sidIdx = 0
pi.primaryBlockMetadata = nil
@@ -61,11 +63,13 @@ func (pi *partIter) reset() {
pi.err = nil
}
-func (pi *partIter) init(p *part, sids []common.SeriesID, minTimestamp, maxTimestamp int64) {
+func (pi *partIter) init(ref partRef, sids []common.SeriesID, minTimestamp, maxTimestamp int64) {
pi.reset()
pi.curBlock = &blockMetadata{}
+ p := ref.p
pi.p = p
- pi.c = p.cache
+ pi.c = ref.cache
+ pi.shardID = ref.shardID
pi.sids = sids
pi.minTimestamp = minTimestamp
diff --git a/banyand/measure/part_iter_test.go b/banyand/measure/part_iter_test.go
index 4a50d5e09..a0679476f 100644
--- a/banyand/measure/part_iter_test.go
+++ b/banyand/measure/part_iter_test.go
@@ -112,10 +112,11 @@ func Test_partIter_nextBlock(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
+ shardCache := storage.NewShardCache("test-group", 0, 0)
verifyPart := func(p *part) {
defer p.close()
pi := partIter{}
- pi.init(p, tt.sids, tt.opt.minTimestamp, tt.opt.maxTimestamp)
+ pi.init(partRef{p: p, cache: shardCache}, tt.sids, tt.opt.minTimestamp, tt.opt.maxTimestamp)
var got []blockMetadata
for pi.nextBlock() {
@@ -147,8 +148,6 @@ func Test_partIter_nextBlock(t *testing.T) {
mp.mustInitFromDataPoints(dps)
p := openMemPart(mp)
- shardCache := storage.NewShardCache("test-group", 0, 0)
- p.cache = shardCache
verifyPart(p)
tmpDir, defFn := test.Space(require.New(t))
defer defFn()
@@ -157,7 +156,6 @@ func Test_partIter_nextBlock(t *testing.T) {
fileSystem := fs.NewLocalFileSystem()
mp.mustFlush(fileSystem, partPath)
p = mustOpenFilePart(epoch, tmpDir, fileSystem)
- p.cache = shardCache
verifyPart(p)
})
}
diff --git a/banyand/measure/part_query_race_test.go b/banyand/measure/part_query_race_test.go
new file mode 100644
index 000000000..291fe5c5e
--- /dev/null
+++ b/banyand/measure/part_query_race_test.go
@@ -0,0 +1,77 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "errors"
+ "sync"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/banyand/internal/storage"
+)
+
+// TestConcurrentQueriesShareParts runs queries with different caches and shard IDs over the
+// same snapshot-owned parts. A query must not write the shared parts: under -race this fails
+// if the cache or shard ID is stored on a part, and the assertions fail if one query observes
+// another's context.
+func TestConcurrentQueriesShareParts(t *testing.T) {
+ fx := newFixture(t, fxPart{{sid: 1, from: 1, to: 50, version: 1}}, fxPart{{sid: 2, from: 1, to: 50, version: 1}})
+ snp := &snapshot{ref: 1}
+ for _, r := range fx.parts {
+ snp.parts = append(snp.parts, newPartWrapper(nil, r.p))
+ }
+ const queries = 16
+ var wg sync.WaitGroup
+ errs := make(chan error, queries)
+ for g := range queries {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ cache := storage.NewShardCache("race-group", 0, 0)
+ shardID := common.ShardID(g)
+ for range 50 {
+ refs, n := snp.getParts(nil, cache, shardID, 0, 1<<40)
+ if n != len(snp.parts) {
+ errs <- errBadPartCount
+ return
+ }
+ ti := generateTstIter()
+ ti.init(refs, fx.sids, 0, 1<<40)
+ for ti.nextBlock() {
+ if pi := ti.piHeap[0]; pi.shardID != shardID || pi.c != cache {
+ errs <- errWrongContext
+ releaseTstIter(ti)
+ return
+ }
+ }
+ releaseTstIter(ti)
+ }
+ }()
+ }
+ wg.Wait()
+ close(errs)
+ require.NoError(t, <-errs)
+}
+
+var (
+ errBadPartCount = errors.New("unexpected part count")
+ errWrongContext = errors.New("query observed another query's cache or shard ID")
+)
diff --git a/banyand/measure/poison_test.go b/banyand/measure/poison_test.go
new file mode 100644
index 000000000..c8e08c5d4
--- /dev/null
+++ b/banyand/measure/poison_test.go
@@ -0,0 +1,23 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+// Unit tests run with released-cursor poisoning on, so a borrowed cell read after
+// its cursor was recycled reads the sentinel instead of stale-but-plausible data,
+// and a double release panics instead of being ignored.
+func init() { setPoisonReleasedCursor(true) }
diff --git a/banyand/measure/query.go b/banyand/measure/query.go
index a5f78eb6d..d28ce94f7 100644
--- a/banyand/measure/query.go
+++ b/banyand/measure/query.go
@@ -175,8 +175,10 @@ func (m *measure) Query(ctx context.Context, mqo model.MeasureQueryOptions) (mqr
segments: segments,
tagProjection: mqo.TagProjection,
storedIndexValue: storedIndexValue,
+ dp: m.decodePool,
}
queryResultTracker.Acquire(&result)
+ queriesStarted.Add(1)
if m.queryMetrics != nil {
m.queryMetrics.totalQueryResultStarted.Inc(1)
}
@@ -198,7 +200,7 @@ func (m *measure) Query(ctx context.Context, mqo model.MeasureQueryOptions) (mqr
}
}
- var parts []*part
+ var parts []partRef
qo := queryOptions{
MeasureQueryOptions: mqo,
schemaTagTypes: schemaTagTypes,
@@ -212,16 +214,11 @@ func (m *measure) Query(ctx context.Context, mqo model.MeasureQueryOptions) (mqr
if s == nil {
continue
}
- oldLen := len(parts)
- parts, n = s.getParts(parts, caches[i], qo.minTimestamp, qo.maxTimestamp)
+ parts, n = s.getParts(parts, caches[i], tableShardIDs[i], qo.minTimestamp, qo.maxTimestamp)
if n < 1 {
s.decRef()
continue
}
- // Set shard ID for newly added parts
- for j := oldLen; j < len(parts); j++ {
- parts[j].shardID = tableShardIDs[i]
- }
result.snapshots = append(result.snapshots, s)
}
@@ -251,11 +248,10 @@ func (m *measure) Query(ctx context.Context, mqo model.MeasureQueryOptions) (mqr
// checks for nil and returns a clean error rather than degrading
// the row-path Pull().
// Thread GroupBy + Agg through so BuildBatchSchema emits native
- // column types for the agg-relevant columns. Plain queries get
- // passthrough; GroupBy+Agg queries get native int64/float64/string
- // columns the BatchAggregation operator reads directly (computeKey +
- // fold). Storage decoders (banyand/measure/batch_decode.go) handle
- // both column shapes.
+ // column types for the agg-relevant columns. Every projected tag and
+ // field whose schema type maps cleanly is a native typed column the
+ // storage decoders (banyand/measure/batch_decode.go) fill without
+ // building protobuf wrappers; only unmapped projections stay passthrough.
result.batchSchema, _ = vmeasure.BuildBatchSchema(m.schema, model.MeasureQueryOptions{
TagProjection: result.tagProjection,
FieldProjection: mqo.FieldProjection,
@@ -594,7 +590,7 @@ func (m *measure) buildIndexQueryResult(ctx context.Context, mqo model.MeasureQu
return r, nil
}
-func (m *measure) searchBlocks(ctx context.Context, result *queryResult, sids []common.SeriesID, parts []*part, qo queryOptions) error {
+func (m *measure) searchBlocks(ctx context.Context, result *queryResult, sids []common.SeriesID, parts []partRef, qo queryOptions) error {
defFn := startBlockScanSpan(ctx, len(sids), parts, result)
defer defFn()
tstIter := generateTstIter()
@@ -609,6 +605,11 @@ func (m *measure) searchBlocks(ctx context.Context, result *queryResult, sids []
var hit int
var totalBlockBytes uint64
quota := m.pm.AvailableBytes()
+ // A scan that streams through the window scan reserves memory block by block
+ // (scan_decode_pool.go), so what it must fit is a block, not its whole result. TopN
+ // and the legacy test switch load every block before they merge, so they keep the
+ // check of the sum.
+ streaming := qo.Name != TopNSchemaName && !legacyBatchScan.Load()
for tstIter.nextBlock() {
if hit%checkDoneEvery == 0 {
select {
@@ -621,18 +622,35 @@ func (m *measure) searchBlocks(ctx context.Context, result *queryResult, sids []
bc := generateBlockCursor()
p := tstIter.piHeap[0]
bc.init(p.p, p.curBlock, qo)
- bc.shardID = p.p.shardID
+ bc.shardID = p.shardID
result.data = append(result.data, bc)
totalBlockBytes += bc.bm.uncompressedSizeBytes
- if quota >= 0 && totalBlockBytes > uint64(quota) {
- return fmt.Errorf("block scan quota exceeded: used %d bytes, quota is %d bytes", totalBlockBytes, quota)
+ if quota >= 0 {
+ // A block larger than the whole quota can never be reserved, streaming or not:
+ // fail before any of it is read. The scans that load everything first also
+ // bound the sum.
+ if used := totalBlockBytes; !streaming && used > uint64(quota) {
+ return fmt.Errorf("block scan quota exceeded: used %d bytes, quota is %d bytes", used, quota)
+ }
+ if size := bc.bm.uncompressedSizeBytes; size > uint64(quota) {
+ return fmt.Errorf("block scan quota exceeded: used %d bytes, quota is %d bytes", size, quota)
+ }
}
}
if tstIter.Error() != nil {
return fmt.Errorf("cannot iterate tstIter: %w", tstIter.Error())
}
- if err := m.pm.AcquireResource(ctx, totalBlockBytes); err != nil {
- return err
+ result.budget = newScanBudget(quota, m.decodePool)
+ result.admit = func(ctx context.Context) error {
+ if quota >= 0 && totalBlockBytes > uint64(quota) {
+ return fmt.Errorf("block scan quota exceeded: used %d bytes, quota is %d bytes", totalBlockBytes, quota)
+ }
+ return m.pm.AcquireResource(ctx, totalBlockBytes)
+ }
+ if !streaming {
+ if err := result.admitWhole(ctx); err != nil {
+ return err
+ }
}
result.sidToIndex = make(map[common.SeriesID]int)
for i, si := range originalSids {
@@ -800,16 +818,36 @@ type queryResult struct {
sidToIndex map[common.SeriesID]int
storedIndexValue map[common.SeriesID]map[string]*modelv1.TagValue
batchSchema *vectorized.BatchSchema
+ scan *windowScan
+ dp *decodePool
+ budget *scanBudget
+ admit func(ctx context.Context) error
tagProjection []model.TagProjection
data []*blockCursor
snapshots []*snapshot
segments []storage.Segment[*tsTable, option]
hit int
loaded bool
+ scanPlanned bool
+ admitted bool
orderByTS bool
ascTS bool
}
+// admitWhole runs the one-shot admission of the whole result, which the paths that
+// load every block before merging them need; the window scan reserves per cursor
+// instead. It runs once.
+func (qr *queryResult) admitWhole(ctx context.Context) error {
+ if qr.admitted || qr.admit == nil {
+ return nil
+ }
+ if err := qr.admit(ctx); err != nil {
+ return err
+ }
+ qr.admitted = true
+ return nil
+}
+
func (qr *queryResult) Pull() *model.MeasureResult {
select {
case <-qr.ctx.Done():
@@ -822,6 +860,9 @@ func (qr *queryResult) Pull() *model.MeasureResult {
if len(qr.data) == 0 {
return nil
}
+ if err := qr.admitWhole(qr.ctx); err != nil {
+ return &model.MeasureResult{Error: err}
+ }
cursorChan := make(chan int, len(qr.data))
measureBlockLogger := logger.GetLogger("measure-query-block-loader")
@@ -842,7 +883,7 @@ func (qr *queryResult) Pull() *model.MeasureResult {
}
tmpBlock := generateBlock()
defer releaseBlock(tmpBlock)
- if !qr.data[idx].loadData(tmpBlock) {
+ if !qr.data[idx].loadData(tmpBlock, false) {
cursorChan <- idx
return
}
@@ -900,6 +941,9 @@ func (qr *queryResult) Release() {
qr.data[i] = nil
}
qr.data = qr.data[:0]
+ if qr.scan != nil {
+ qr.scan.close()
+ }
for i := range qr.snapshots {
qr.snapshots[i].decRef()
}
diff --git a/banyand/measure/query_batch.go b/banyand/measure/query_batch.go
index 9ba8de365..aba2c8af1 100644
--- a/banyand/measure/query_batch.go
+++ b/banyand/measure/query_batch.go
@@ -61,7 +61,7 @@ var (
// "interrupt: blank/total=%d/%d" form. PullBatch propagates these as
// errors instead of as *MeasureResult{Error: …} — its return type is
// (*MeasureBatch, error).
-func (qr *queryResult) PullBatch(_ context.Context) (*model.MeasureBatch, error) {
+func (qr *queryResult) PullBatch(ctx context.Context) (*model.MeasureBatch, error) {
if qr.batchSchema == nil {
return nil, fmt.Errorf("queryResult.PullBatch: batchSchema not initialized; " +
"the underlying query did not record a vectorized BatchSchema (likely a schema-build error at Query time)")
@@ -71,6 +71,9 @@ func (qr *queryResult) PullBatch(_ context.Context) (*model.MeasureBatch, error)
return nil, errors.WithMessagef(qr.ctx.Err(), "interrupt: hit %d", qr.hit)
default:
}
+ if qr.useWindowScan() {
+ return qr.scan.next(ctx)
+ }
if loadErr := qr.loadCursorsForBatch(); loadErr != nil {
return nil, loadErr
}
@@ -102,6 +105,25 @@ func (qr *queryResult) PullBatch(_ context.Context) (*model.MeasureBatch, error)
return qr.mergeBatch(qr.storedIndexValue, qr.batchSchema)
}
+// useWindowScan reports whether PullBatch streams series windows, planning the
+// scan on first use. TopN queries (their merge rewrites field payloads) keep the
+// legacy load-then-merge path below, as does everything while the test switch is
+// on; mergeBatch and loadCursorsForBatch are reachable only through those two.
+func (qr *queryResult) useWindowScan() bool {
+ if qr.scan != nil {
+ return true
+ }
+ if qr.scanPlanned {
+ return false
+ }
+ qr.scanPlanned = true
+ if qr.loaded || qr.topNQueryOptions != nil || legacyBatchScan.Load() || len(qr.data) == 0 {
+ return false
+ }
+ qr.scan = newWindowScan(qr)
+ return true
+}
+
// mergeBatch is the batch-aware counterpart of queryResult.merge (query.go).
// It consumes the heap with the same semantics — version-replacement for
// duplicate timestamps within the same series, series-boundary stop — but
@@ -192,6 +214,13 @@ func (qr *queryResult) loadCursorsForBatch() error {
if len(qr.data) == 0 {
return nil
}
+ if err := qr.admitWhole(qr.ctx); err != nil {
+ return err
+ }
+ // Typed numeric columns are understood by the batch readers only. TopN
+ // queries fall back to the row readers (merge / mergeTopNResult), which
+ // need the bytes form.
+ typedNumeric := qr.topNQueryOptions == nil
cursorChan := make(chan int, len(qr.data))
for i := 0; i < len(qr.data); i++ {
go func(i int) {
@@ -203,7 +232,7 @@ func (qr *queryResult) loadCursorsForBatch() error {
}
tmpBlock := generateBlock()
defer releaseBlock(tmpBlock)
- if !qr.data[i].loadData(tmpBlock) {
+ if !qr.data[i].loadData(tmpBlock, typedNumeric) {
cursorChan <- i
return
}
diff --git a/banyand/measure/query_batch_top_test.go b/banyand/measure/query_batch_top_test.go
index 487c17a93..3aebb3001 100644
--- a/banyand/measure/query_batch_top_test.go
+++ b/banyand/measure/query_batch_top_test.go
@@ -37,6 +37,7 @@ import (
// Widening this gate to fire on a client Top clause would silently put ordinary
// multi-block Top queries back on the row merge, which is what this asserts against.
func TestApplyTopNOptionsOnlyForTopNSchema(t *testing.T) {
+ verifyNoScanLeaks(t)
tests := []struct {
name string
measureName string
@@ -69,6 +70,7 @@ func TestApplyTopNOptionsOnlyForTopNSchema(t *testing.T) {
// spanning more than one block cursor and asserts it merges natively, returning
// every row in timestamp order without the row-path detour.
func TestPullBatchMultiBlockStaysBatchNative(t *testing.T) {
+ verifyNoScanLeaks(t)
h := setupBenchStorageHarness(t)
var qr queryResult
@@ -107,8 +109,9 @@ func TestPullBatchMultiBlockStaysBatchNative(t *testing.T) {
break
}
if !checkedMultiBlock {
- require.Greater(t, len(qr.data), 1,
- "the merge must run over more than one loaded cursor, not the single-block fast path")
+ require.NotNil(t, qr.scan, "a time-ordered query is served by the window scan")
+ require.Greater(t, qr.scan.stats.mergeCursors, int64(1),
+ "the merge must run over more than one decoded cursor, not the single-block fast path")
checkedMultiBlock = true
}
require.NotEmpty(t, batch.Timestamps, "a non-nil batch must carry rows")
diff --git a/banyand/measure/query_test.go b/banyand/measure/query_test.go
index c72a19719..a0f60120e 100644
--- a/banyand/measure/query_test.go
+++ b/banyand/measure/query_test.go
@@ -46,6 +46,7 @@ import (
)
func TestQueryResult(t *testing.T) {
+ verifyNoScanLeaks(t)
tests := []struct {
wantErr error
name string
@@ -1247,7 +1248,7 @@ func TestQueryResult(t *testing.T) {
require.NotNil(t, s)
defer s.decRef()
shardCache := storage.NewShardCache("test-group", 0, 0)
- pp, _ := s.getParts(nil, shardCache, queryOpts.minTimestamp, queryOpts.maxTimestamp)
+ pp, _ := s.getParts(nil, shardCache, 0, queryOpts.minTimestamp, queryOpts.maxTimestamp)
sids := make([]common.SeriesID, len(tt.sids))
copy(sids, tt.sids)
sort.Slice(sids, func(i, j int) bool {
@@ -1379,6 +1380,7 @@ func allPartsFlushed(tst *tsTable) func() bool {
}
func TestQueryResult_QuotaExceeded(t *testing.T) {
+ verifyNoScanLeaks(t)
tests := []struct {
wantErr error
name string
@@ -1480,13 +1482,16 @@ func TestQueryResult_QuotaExceeded(t *testing.T) {
require.NotNil(t, s)
defer s.decRef()
shardCache := storage.NewShardCache("test-group", 0, 0)
- pp, _ := s.getParts(nil, shardCache, queryOpts.minTimestamp, queryOpts.maxTimestamp)
+ pp, _ := s.getParts(nil, shardCache, 0, queryOpts.minTimestamp, queryOpts.maxTimestamp)
var result queryResult
result.ctx = context.TODO()
// Query all tags
result.tagProjection = allTagProjections
err = m.searchBlocks(context.TODO(), &result, tt.sids, pp, queryOpts)
if tt.expectQuotaExceeded {
+ // Query releases the result on a searchBlocks error; this test calls
+ // searchBlocks directly, so it releases the cursors already collected.
+ defer result.Release()
require.Error(t, err)
require.Contains(t, err.Error(), "quota exceeded", "expected quota to be exceeded but got: %v", err)
return
@@ -1523,6 +1528,7 @@ func TestQueryResult_QuotaExceeded(t *testing.T) {
}
func TestSegResultHeap_Sorting(t *testing.T) {
+ verifyNoScanLeaks(t)
tests := []struct {
name string
segResults []*segResult
@@ -1731,6 +1737,7 @@ func TestSegResultHeap_Sorting(t *testing.T) {
}
func TestSegResultHeap_NPE_Prevention(t *testing.T) {
+ verifyNoScanLeaks(t)
tests := []struct {
name string
segResults []*segResult
@@ -1796,6 +1803,7 @@ func TestSegResultHeap_NPE_Prevention(t *testing.T) {
}
func TestIndexSortResult_OrderBySortDesc(t *testing.T) {
+ verifyNoScanLeaks(t)
tests := []struct {
name string
sortOrder modelv1.Sort
diff --git a/banyand/measure/race_off_test.go b/banyand/measure/race_off_test.go
new file mode 100644
index 000000000..1cd7383e7
--- /dev/null
+++ b/banyand/measure/race_off_test.go
@@ -0,0 +1,29 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build !race
+
+package measure
+
+// raceEnabled reports whether the test binary is built with the race detector.
+const raceEnabled = false
+
+// frontierSeries is the series count of the frontier memory-bound tests.
+const frontierSeries = 40
+
+// mixedTailEnd is the last timestamp of the long series of mixedFixture. The long series of mixedFixture spans several blocks (8,192 rows each).
+const mixedTailEnd = 20000
diff --git a/banyand/measure/race_on_test.go b/banyand/measure/race_on_test.go
new file mode 100644
index 000000000..3b3751f01
--- /dev/null
+++ b/banyand/measure/race_on_test.go
@@ -0,0 +1,30 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build race
+
+package measure
+
+// raceEnabled reports whether the test binary is built with the race detector.
+const raceEnabled = true
+
+// frontierSeries is the series count of the frontier memory-bound tests: the race
+// build keeps it large enough for a frontier of several blocks per quarter.
+const frontierSeries = 16
+
+// mixedTailEnd is the last timestamp of the long series of mixedFixture. The race build shortens the long series of mixedFixture to just over one block (8,192 rows): still several blocks, a tail part overlapping its last block, and pages cut at 4,096 rows.
+const mixedTailEnd = 8400
diff --git a/banyand/measure/scan_concurrent_benchmark_test.go b/banyand/measure/scan_concurrent_benchmark_test.go
new file mode 100644
index 000000000..7c6dbb1cc
--- /dev/null
+++ b/banyand/measure/scan_concurrent_benchmark_test.go
@@ -0,0 +1,62 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "sync"
+ "testing"
+)
+
+// BenchmarkVectorizedStorageConcurrent runs the W2, W9, W10 and W11 shapes with
+// several queries at once, the way a data node serves concurrent dashboards. One
+// op is a round of n queries started together; the decode pool is shared by all
+// of them.
+func BenchmarkVectorizedStorageConcurrent(b *testing.B) {
+ hs := newGateHarnesses(b)
+ for _, w := range gateWorkloads() {
+ switch w.id {
+ case "W2", "W9", "W11":
+ default:
+ continue
+ }
+ for _, n := range []int{1, 8} {
+ b.Run(w.id+"x"+string(rune('0'+n)), func(b *testing.B) {
+ h := hs.get(w.layout)
+ b.ReportAllocs()
+ b.ResetTimer()
+ sink := 0
+ var mu sync.Mutex
+ for b.Loop() {
+ var wg sync.WaitGroup
+ for range n {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ rows := runGateWorkload(h, w, false)
+ mu.Lock()
+ sink += rows
+ mu.Unlock()
+ }()
+ }
+ wg.Wait()
+ }
+ gateSink = sink
+ })
+ }
+ }
+}
diff --git a/banyand/measure/scan_const_test.go b/banyand/measure/scan_const_test.go
new file mode 100644
index 000000000..4e4447753
--- /dev/null
+++ b/banyand/measure/scan_const_test.go
@@ -0,0 +1,438 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "fmt"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+ vmeasure "github.com/apache/skywalking-banyandb/pkg/query/vectorized/measure"
+)
+
+// withFastPaths runs fn with the ConstColumn and runs fast paths on (true) or off.
+func withFastPaths(on bool, fn func()) {
+ prevConst := disableConstColumns.Swap(!on)
+ prevRuns := vmeasure.SetRunsDisabled(!on)
+ defer func() {
+ disableConstColumns.Store(prevConst)
+ vmeasure.SetRunsDisabled(prevRuns)
+ }()
+ fn()
+}
+
+// pullAll drains qr batch by batch through PullBatch, handing each to inspect.
+func pullAll(t *testing.T, qr *queryResult, inspect func(b *model.MeasureBatch)) {
+ t.Helper()
+ for {
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ break
+ }
+ inspect(b)
+ b.Release()
+ }
+ qr.Release()
+}
+
+func constOf(t *testing.T, col vectorized.Column) *vectorized.ConstColumn {
+ t.Helper()
+ c, ok := vectorized.AsConst(col)
+ require.True(t, ok, "expected a ConstColumn, got %T", col)
+ return c
+}
+
+func TestScanWindowLendsEntityTagsAsConstColumns(t *testing.T) {
+ verifyNoScanLeaks(t)
+ // Series 1 and 2 are large and disjoint in time (lent blocks); 3 and 4 are small
+ // and disjoint from everything (coalesced into one batch of two series).
+ fx := newFixture(t, fxPart{{1, 1, 3000, 1}, {2, 3001, 6000, 1}, {3, 7000, 7010, 1}, {4, 7020, 7032, 1}})
+ stored := map[common.SeriesID]map[string]*modelv1.TagValue{
+ 1: {fxTagHid: strTagValue("hidden-1")},
+ 2: {fxTagHid: strTagValue("hidden-2")},
+ }
+ q := &fxQuery{
+ tags: []string{fxTagSvc, fxTagURL, fxTagHid, fxTagN2}, fields: []string{fxFldInt}, minTS: 0, maxTS: 10000,
+ stored: stored, orderByTS: true, ascTS: true,
+ }
+ schema := q.schema(t)
+ svcIdx, urlIdx, hidIdx, n2Idx := 0, 1, 2, 3
+ var lent, pages int
+ withFastPaths(true, func() {
+ pullAll(t, fx.newResult(t, q, schema), func(b *model.MeasureBatch) {
+ require.True(t, b.SeriesKnown)
+ require.Equal(t, vectorized.TimeOrderAsc, b.TimeOrder)
+ require.Positive(t, b.ActiveRows())
+ // Series boundaries are exact: runs of equal series ids.
+ var want []int
+ for i := 1; i < b.RowCount(); i++ {
+ if b.SeriesIDs[i] != b.SeriesIDs[i-1] {
+ want = append(want, i)
+ }
+ }
+ if len(want) > 0 {
+ want = append(want, b.RowCount())
+ }
+ require.Equal(t, want, append([]int(nil), b.SeriesBoundaries...))
+ // A column that is not an entity tag is never constant.
+ _, isConst := vectorized.AsConst(b.Tags[urlIdx])
+ require.False(t, isConst)
+ _, isConst = vectorized.AsConst(b.Tags[n2Idx])
+ require.False(t, isConst)
+ if len(want) > 0 {
+ // Rows of several series stay typed: no single constant covers them.
+ pages++
+ _, isConst = vectorized.AsConst(b.Tags[svcIdx])
+ require.False(t, isConst, "an interleaved page is not constant")
+ return
+ }
+ lent++
+ sid := b.SeriesIDs[0]
+ svc := constOf(t, b.Tags[svcIdx])
+ require.Equal(t, fmt.Sprintf("svc-%d", int64(sid)%4), svc.Str())
+ require.Equal(t, b.RowCount(), svc.Len())
+ hid := constOf(t, b.Tags[hidIdx])
+ if tv, ok := stored[sid]; ok {
+ require.Equal(t, tv[fxTagHid].GetStr().GetValue(), hid.Str(), "an index-substituted entity tag is constant per series")
+ } else {
+ require.True(t, hid.IsNullValue(), "a tag no row carries is a null constant")
+ }
+ })
+ })
+ require.Positive(t, lent)
+ require.Positive(t, pages, "the two small series are coalesced into one multi-series batch")
+
+ // With the fast paths off the same query produces no constant at all.
+ withFastPaths(false, func() {
+ pullAll(t, fx.newResult(t, q, schema), func(b *model.MeasureBatch) {
+ for _, col := range b.Tags {
+ _, isConst := vectorized.AsConst(col)
+ require.False(t, isConst)
+ }
+ })
+ })
+}
+
+func TestScanWindowCoalescedSingleSeriesBatchCarriesConstEntityTags(t *testing.T) {
+ verifyNoScanLeaks(t)
+ // Several small segments of one series are coalesced into one owned batch.
+ fx := newFixture(t,
+ fxPart{{1, 1, 40, 1}}, fxPart{{1, 100, 140, 1}}, fxPart{{1, 200, 240, 1}}, fxPart{{1, 300, 340, 1}},
+ )
+ q := &fxQuery{tags: []string{fxTagSvc, fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 10000, ascTS: true}
+ schema := q.schema(t)
+ var coalesced int
+ withFastPaths(true, func() {
+ qr := fx.newResult(t, q, schema)
+ pullAll(t, qr, func(b *model.MeasureBatch) {
+ require.True(t, b.SeriesKnown)
+ require.Empty(t, b.SeriesBoundaries)
+ svc := constOf(t, b.Tags[0])
+ require.Equal(t, "svc-1", svc.Str())
+ require.Equal(t, b.RowCount(), svc.Len())
+ coalesced++
+ })
+ })
+ require.Equal(t, 1, coalesced)
+}
+
+func TestScanWindowDescendingBatchesAreHintedDescending(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := newFixture(t, fxPart{{1, 1, 3000, 1}, {2, 4000, 6000, 1}})
+ q := &fxQuery{tags: []string{fxTagSvc}, fields: []string{fxFldInt}, minTS: 0, maxTS: 10000, orderByTS: true, ascTS: false}
+ schema := q.schema(t)
+ withFastPaths(true, func() {
+ pullAll(t, fx.newResult(t, q, schema), func(b *model.MeasureBatch) {
+ require.Equal(t, vectorized.TimeOrderDesc, b.TimeOrder)
+ require.NotNil(t, b.Selection, "a lent descending block reads through a reversed selection")
+ ts := b.Timestamps
+ for i := 1; i < len(b.Selection); i++ {
+ require.GreaterOrEqual(t, ts[b.Selection[i-1]], ts[b.Selection[i]])
+ }
+ constOf(t, b.Tags[0])
+ })
+ })
+}
+
+// TestScanWindowConstAndRunsMatchPerRowAndLegacy runs queries three ways: legacy scan,
+// window scan with the ConstColumn and runs fast paths off, and with them on. Group-by
+// on entity tags, time buckets over many series, and the OAP multi-series shapes must
+// agree row for row and byte for byte.
+func TestScanWindowConstAndRunsMatchPerRowAndLegacy(t *testing.T) {
+ verifyNoScanLeaks(t)
+ oapSeries, staggeredSeries := 40, 12
+ if raceEnabled {
+ // Still more than ten series (the fifth part's reach) and spanning every parity
+ // and multiple-of-five rewrite the oap parts are built from.
+ oapSeries, staggeredSeries = 12, 8
+ }
+ fxNames := []string{"oap", "staggered", "mixed"}
+ fixtures := map[string]*fxFixture{"oap": oapFixture(t, oapSeries), "staggered": staggeredFixture(t, staggeredSeries), "mixed": mixedFixture(t)}
+ hidden := map[common.SeriesID]map[string]*modelv1.TagValue{}
+ for sid := common.SeriesID(1); sid <= 40; sid++ {
+ hidden[sid] = map[string]*modelv1.TagValue{fxTagHid: strTagValue(fmt.Sprintf("hid-%d", sid%3))}
+ }
+ byTag := func(names ...string) *model.MeasureGroupBy {
+ return &model.MeasureGroupBy{TagFamily: fxFamily, TagNames: names}
+ }
+ bucket := func(width int64, names ...string) *model.MeasureGroupBy {
+ gb := &model.MeasureGroupBy{TimeBucket: &model.MeasureTimeBucket{Width: fmt.Sprintf("%dns", width), WidthNanos: width}}
+ if len(names) > 0 {
+ gb.TagFamily, gb.TagNames = fxFamily, names
+ }
+ return gb
+ }
+ aggs := []*model.MeasureAgg{
+ {FieldName: fxFldInt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_SUM},
+ {FieldName: fxFldFlt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_SUM},
+ {FieldName: fxFldFlt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_MEAN},
+ {FieldName: fxFldInt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_MIN},
+ {FieldName: fxFldNul, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_COUNT},
+ }
+ groupNames := []string{
+ "scalar", "entity tag", "hidden entity tag", "both entity tags", "bucket", "bucket + entity tag", "wide bucket + entity tags",
+ }
+ groupings := map[string]*model.MeasureGroupBy{
+ "scalar": nil,
+ "entity tag": byTag(fxTagSvc),
+ "hidden entity tag": byTag(fxTagHid),
+ "both entity tags": byTag(fxTagSvc, fxTagHid),
+ "bucket": bucket(64),
+ "bucket + entity tag": bucket(64, fxTagSvc),
+ "wide bucket + entity tags": bucket(1000, fxTagSvc, fxTagHid),
+ }
+ orderNames := []string{"no time order", "time asc", "time desc"}
+ orders := map[string]func(q *fxQuery){
+ "no time order": func(_ *fxQuery) {},
+ "time asc": func(q *fxQuery) { q.orderByTS, q.ascTS = true, true },
+ "time desc": func(q *fxQuery) { q.orderByTS, q.ascTS = true, false },
+ }
+ // combos are [fixture, grouping, order, aggregation] indexes of the valid cross product.
+ var combos [][]int
+ for f := range fxNames {
+ for g, gName := range groupNames {
+ gb := groupings[gName]
+ for o, oName := range orderNames {
+ for a, agg := range aggs {
+ if gb == nil && agg.FieldName == fxFldNul {
+ continue
+ }
+ if gb != nil && gb.TimeBucket != nil && oName != "time asc" {
+ // A streaming time bucket needs a time-ascending input (the planner
+ // does not stream any other).
+ continue
+ }
+ combos = append(combos, []int{f, g, o, a})
+ }
+ }
+ }
+ }
+ if raceEnabled {
+ // Every fixture, grouping, order and aggregation still runs, and so does every
+ // (grouping, aggregation) and (grouping, order) pair; the costly mixed fixture is picked last.
+ combos = coverPairs(combos, [][2]int{{1, 3}, {1, 2}}, func(c []int) int { return []int{3, 2, 12}[c[0]] })
+ }
+ for _, c := range combos {
+ fxName, gName, oName, agg := fxNames[c[0]], groupNames[c[1]], orderNames[c[2]], aggs[c[3]]
+ fx, gb, order := fixtures[fxName], groupings[gName], orders[oName]
+ q := &fxQuery{
+ tags: []string{fxTagSvc, fxTagHid}, fields: []string{fxFldInt, fxFldFlt, fxFldNul}, minTS: 0, maxTS: 1 << 20,
+ groupBy: gb, agg: agg, stored: hidden,
+ }
+ order(q)
+ what := fmt.Sprintf("%s/%s/%s/%s(%s)", fxName, gName, oName, agg.Func, agg.FieldName)
+ t.Run(what, func(t *testing.T) {
+ legacy := fx.run(t, q, true)
+ var plain, fast fxResult
+ withFastPaths(false, func() { plain = fx.run(t, q, false) })
+ withFastPaths(true, func() { fast = fx.run(t, q, false) })
+ requireSameResult(t, legacy, plain, what+" per-row vs legacy")
+ requireSameResult(t, plain, fast, what+" fast vs per-row")
+ })
+ }
+}
+
+// TestScanWindowRawQueriesMatchWithConstColumns covers the non-aggregating egress
+// (rows and raw frames) over entity tags carried as constants.
+func TestScanWindowRawQueriesMatchWithConstColumns(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := mixedFixture(t)
+ hidden := map[common.SeriesID]map[string]*modelv1.TagValue{1: {fxTagHid: strTagValue("hidden-1")}, 6: {fxTagHid: strTagValue("hidden-6")}}
+ lims := [][2]uint32{{5, 70}, {0, 0}}
+ var combos [][]int
+ for ordered := range 2 {
+ for asc := range 2 {
+ for l := range lims {
+ combos = append(combos, []int{ordered, asc, l})
+ }
+ }
+ }
+ if raceEnabled {
+ // The unbounded limit returns every row of the mixed fixture: it is costly, so
+ // the cover takes it as rarely as the pairs allow.
+ combos = coverPairs(combos, [][2]int{{0, 1}}, func(c []int) int { return 1 + 10*c[2] })
+ }
+ for _, c := range combos {
+ ordered, asc, lim := c[0] == 1, c[1] == 0, lims[c[2]]
+ q := &fxQuery{
+ tags: []string{fxTagSvc, fxTagHid, fxTagURL, fxTagN}, fields: []string{fxFldInt, fxFldStr}, minTS: 0, maxTS: 30000,
+ stored: hidden, orderByTS: ordered, ascTS: asc, offset: lim[0], limit: lim[1],
+ }
+ what := fmt.Sprintf("ordered=%v asc=%v limit=%v", ordered, asc, lim)
+ t.Run(what, func(t *testing.T) {
+ legacy := fx.run(t, q, true)
+ var fast fxResult
+ withFastPaths(true, func() { fast = fx.run(t, q, false) })
+ requireSameResult(t, legacy, fast, what)
+ })
+ }
+}
+
+// TestScanWindowEntityTagFilterKeepsOrDropsWholeBatches filters a scan on an entity tag
+// and compares the rows with the same filter over typed columns.
+func TestScanWindowEntityTagFilterKeepsOrDropsWholeBatches(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := oapFixture(t, 24)
+ q := &fxQuery{tags: []string{fxTagSvc, fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 20, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ keep := func(col vectorized.Column, row int) bool {
+ if c, ok := vectorized.AsConst(col); ok {
+ return c.Str() == "svc-1" || c.Str() == "svc-3"
+ }
+ v := col.(*vectorized.TypedColumn[string]).Data()[row]
+ return v == "svc-1" || v == "svc-3"
+ }
+ run := func(on bool) (rows string, filtered, wholeDrops int) {
+ withFastPaths(on, func() {
+ qr := fx.newResult(t, q, schema)
+ pool := vectorizedBatchPool(schema)
+ source := newTestBatchSource(qr, schema, pool)
+ defer source.Close()
+ filter := vmeasure.NewBatchColumnFilter(schema, tagCol(schema, fxTagSvc), keep)
+ for {
+ b, err := source.NextBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ return
+ }
+ require.NoError(t, filter.Process(context.Background(), b))
+ filtered++
+ if b.ActiveLen() == 0 {
+ wholeDrops++
+ }
+ for _, r := range activeRows(b) {
+ rows += fmt.Sprintf("%v/%v/%v;", b.Columns[schema.TimestampIndex()].(*vectorized.TypedColumn[int64]).Data()[r],
+ colString(b.Columns[tagCol(schema, fxTagSvc)], r), colString(b.Columns[tagCol(schema, fxTagURL)], r))
+ }
+ pool.Put(b)
+ }
+ })
+ return
+ }
+ wantRows, _, _ := run(false)
+ gotRows, filtered, wholeDrops := run(true)
+ require.NotEmpty(t, wantRows)
+ require.Equal(t, wantRows, gotRows)
+ require.Positive(t, filtered)
+ require.Positive(t, wholeDrops, "batches of series the filter rejects are dropped whole")
+}
+
+func tagCol(schema *vectorized.BatchSchema, name string) int {
+ idx, ok := schema.TagIndex(fxFamily, name)
+ if !ok {
+ panic("no tag column " + name)
+ }
+ return idx
+}
+
+func activeRows(b *vectorized.RecordBatch) []int {
+ if b.Selection == nil {
+ rows := make([]int, b.Len)
+ for i := range rows {
+ rows[i] = i
+ }
+ return rows
+ }
+ rows := make([]int, len(b.Selection))
+ for i, r := range b.Selection {
+ rows[i] = int(r)
+ }
+ return rows
+}
+
+func colString(col vectorized.Column, row int) string {
+ if c, ok := vectorized.AsConst(col); ok {
+ return c.Expand().(*vectorized.TypedColumn[string]).Data()[0]
+ }
+ return col.(*vectorized.TypedColumn[string]).Data()[row]
+}
+
+func TestScanWindowMergePagesKeepTypedEntityColumns(t *testing.T) {
+ verifyNoScanLeaks(t)
+ // Two overlapping blocks of one series are merged: the page is one series but is
+ // not lent from one cursor, so its entity tags stay typed (the hints still apply).
+ fx := newFixture(t, fxPart{{1, 1, 600, 1}}, fxPart{{1, 300, 900, 2}})
+ q := &fxQuery{tags: []string{fxTagSvc}, fields: []string{fxFldInt}, minTS: 0, maxTS: 10000, ascTS: true}
+ schema := q.schema(t)
+ var pages int
+ withFastPaths(true, func() {
+ pullAll(t, fx.newResult(t, q, schema), func(b *model.MeasureBatch) {
+ pages++
+ require.True(t, b.SeriesKnown)
+ _, isConst := vectorized.AsConst(b.Tags[0])
+ require.False(t, isConst)
+ })
+ })
+ require.Positive(t, pages)
+}
+
+// TestScanWindowEntityTagOfAnotherStoredTypeIsAConstantNull checks the schema-type
+// mismatch case: a column stored as another type than the schema declares reads as
+// null, row by row on the legacy path, so a lent batch carries it as a null constant.
+func TestScanWindowEntityTagOfAnotherStoredTypeIsAConstantNull(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := newFixture(t, fxPart{{1, 1, 600, 1}, {2, 1, 600, 1}})
+ declared := fxTagTypes[fxTagSvc]
+ fxTagTypes[fxTagSvc] = pbv1.ValueTypeInt64
+ defer func() { fxTagTypes[fxTagSvc] = declared }()
+
+ q := &fxQuery{tags: []string{fxTagSvc, fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 10000, ascTS: true}
+ schema := q.schema(t)
+ var lent int
+ withFastPaths(true, func() {
+ pullAll(t, fx.newResult(t, q, schema), func(b *model.MeasureBatch) {
+ lent++
+ svc := constOf(t, b.Tags[0])
+ require.True(t, svc.IsNullValue(), "a stored type that differs from the schema's reads as null")
+ require.Equal(t, b.RowCount(), svc.Len())
+ })
+ })
+ require.Positive(t, lent)
+ legacy := fx.run(t, q, true)
+ var fast fxResult
+ withFastPaths(true, func() { fast = fx.run(t, q, false) })
+ requireSameResult(t, legacy, fast, "stored type differs from the schema")
+}
diff --git a/banyand/measure/scan_cover_test.go b/banyand/measure/scan_cover_test.go
new file mode 100644
index 000000000..576aa4578
--- /dev/null
+++ b/banyand/measure/scan_cover_test.go
@@ -0,0 +1,86 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+// coverPairs deterministically selects a subset of combos (each a tuple of dimension
+// value indexes) such that every value of every dimension and every value pair of the
+// dimension pairs listed in pairs appears in at least one selected combo. A nil pairs
+// requires every pair of dimensions. The selection is greedy and ordered, so equal
+// input always yields equal output; the race build of the differential tests runs it
+// instead of the full cross product. A non-nil cost ranks combos by how expensive they
+// are to run (at least 1): the greedy step takes the best gain per unit of cost, so
+// the cheap combos that cover a value are preferred over the heavy ones.
+func coverPairs(combos [][]int, pairs [][2]int, cost func(combo []int) int) [][]int {
+ if len(combos) == 0 {
+ return nil
+ }
+ dims := len(combos[0])
+ if pairs == nil {
+ for i := range dims {
+ for j := i + 1; j < dims; j++ {
+ pairs = append(pairs, [2]int{i, j})
+ }
+ }
+ }
+ type item struct{ a, b, av, bv int }
+ items := func(c []int) []item {
+ out := make([]item, 0, dims+len(pairs))
+ for d := range dims {
+ out = append(out, item{d, -1, c[d], -1})
+ }
+ for _, p := range pairs {
+ out = append(out, item{p[0], p[1], c[p[0]], c[p[1]]})
+ }
+ return out
+ }
+ missing := map[item]struct{}{}
+ for _, c := range combos {
+ for _, it := range items(c) {
+ missing[it] = struct{}{}
+ }
+ }
+ var picked [][]int
+ used := make([]bool, len(combos))
+ for len(missing) > 0 {
+ best, bestScore := -1, 0.0
+ for i, c := range combos {
+ if used[i] {
+ continue
+ }
+ gain := 0
+ for _, it := range items(c) {
+ if _, ok := missing[it]; ok {
+ gain++
+ }
+ }
+ weight := 1
+ if cost != nil {
+ weight = max(1, cost(c))
+ }
+ if score := float64(gain) / float64(weight); score > bestScore {
+ best, bestScore = i, score
+ }
+ }
+ used[best] = true
+ picked = append(picked, combos[best])
+ for _, it := range items(combos[best]) {
+ delete(missing, it)
+ }
+ }
+ return picked
+}
diff --git a/banyand/measure/scan_decode_pool.go b/banyand/measure/scan_decode_pool.go
new file mode 100644
index 000000000..7d5de3237
--- /dev/null
+++ b/banyand/measure/scan_decode_pool.go
@@ -0,0 +1,396 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "fmt"
+ "runtime/debug"
+ "sync"
+ "sync/atomic"
+ "time"
+
+ "github.com/pkg/errors"
+
+ "github.com/apache/skywalking-banyandb/banyand/protector"
+ "github.com/apache/skywalking-banyandb/pkg/logger"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+// Node-wide decode pool and scan memory accounting (measure scan path v2, step 4b).
+//
+// One decodePool serves every query of a data node. A scan asks it to decode the
+// cursors it will consume next (scan_prefetch.go); a worker runs the same loadData
+// path the scan would run inline, so the pool changes when a block is decoded, never
+// what is decoded. A pool per query would bring back the per-block fan-out the
+// window scan removed as soon as queries run concurrently.
+//
+// Memory is accounted per cursor. Before a cursor is decoded the scan reserves the
+// block's uncompressed size, known from block metadata, from the query's budget;
+// the reservation is attached to the cursor and released when its last reference
+// goes, that is after the scan and every batch that borrows its buffers are done
+// with it. A buffer pinned by several batches is therefore counted once.
+//
+// protector.Memory.AcquireResource, which the one-shot check used to call for the
+// whole result, is an admission gate: it waits until sampled process usage leaves
+// room and holds nothing, so there is nothing to release. The reservation counters
+// here are what the scan can account incrementally; the protector is still asked
+// for headroom, and still has the last word.
+
+var (
+ errDecodePoolClosed = errors.New("measure decode pool is closed")
+ errScanClosed = errors.New("measure scan is closed")
+)
+
+type taskState = int32
+
+const (
+ taskQueued taskState = iota
+ taskRunning
+ taskFinished
+ // taskCanceling is the instant between winning the cancellation and publishing its
+ // error; the task counts as done only once it is taskCanceled.
+ taskCanceling
+ taskCanceled
+)
+
+// decodeTask is one cursor decode queued on a decodePool. It is owned by the
+// prefetcher that created it; the pool only runs it.
+type decodeTask struct {
+ enqueued time.Time
+ ctx context.Context
+ load cursorLoader
+ bc *blockCursor
+ qm *queryMetrics
+ // wake is signaled each time a task of the scan finishes or is canceled; one
+ // channel serves the scan's tasks, whose consumer re-checks the one it waits for.
+ wake chan struct{}
+ err error
+ state atomic.Int32
+ ok bool
+ desc bool
+}
+
+func (t *decodeTask) init(ctx context.Context, load cursorLoader, bc *blockCursor, desc bool, qm *queryMetrics, wake chan struct{}) {
+ t.ctx, t.load, t.bc, t.desc, t.qm, t.wake = ctx, load, bc, desc, qm, wake
+ t.enqueued = time.Now()
+ liveDecodeTasks.Add(1)
+}
+
+// clear drops the references of a task that has ended and whose result was read, or
+// that was withdrawn before it started. It leaves the state, which a stale queue
+// entry may still check, so such an entry stays a no-op.
+func (t *decodeTask) clear() {
+ t.ctx, t.load, t.bc, t.qm, t.wake, t.err = nil, nil, nil, nil, nil, nil
+}
+
+// finished reports whether the task has ended, by running or by cancellation; its
+// result is then readable.
+func (t *decodeTask) finished() bool {
+ s := t.state.Load()
+ return s == taskFinished || s == taskCanceled
+}
+
+// signal wakes the consumer through wake, which the caller read while it owned the
+// task, before it published the task's end: once the end is visible the consumer may clear the task.
+func signal(wake chan struct{}) {
+ select {
+ case wake <- struct{}{}:
+ default:
+ }
+}
+
+// cancel stops a task that has not started. It reports whether it did; a task
+// that is running or finished is left alone and ends by itself.
+func (t *decodeTask) cancel(err error) bool {
+ if !t.state.CompareAndSwap(taskQueued, taskCanceling) {
+ return false
+ }
+ wake := t.wake
+ t.err = err
+ t.state.Store(taskCanceled)
+ liveDecodeTasks.Add(-1)
+ signal(wake)
+ return true
+}
+
+func (t *decodeTask) run() {
+ if !t.state.CompareAndSwap(taskQueued, taskRunning) {
+ return
+ }
+ wake := t.wake
+ if t.qm != nil {
+ t.qm.scanDecodeQueueWait.Observe(time.Since(t.enqueued).Seconds())
+ }
+ defer func() {
+ if r := recover(); r != nil {
+ t.ok = false
+ t.err = fmt.Errorf("panic while decoding a block: %v", r)
+ logger.GetLogger("measure-decode-pool").Error().Interface("panic", r).
+ Bytes("stack", debug.Stack()).Msg("panic in the decode pool")
+ }
+ t.state.Store(taskFinished)
+ liveDecodeTasks.Add(-1)
+ signal(wake)
+ }()
+ if err := t.ctx.Err(); err != nil {
+ t.err = err
+ return
+ }
+ t.ok, t.err = t.load(t.ctx, t.bc, t.desc)
+}
+
+// decodePool is the node-wide set of decode workers and the node's total of scan
+// reservations.
+type decodePool struct {
+ pm protector.Memory
+ cond *sync.Cond
+ // urgent holds the decodes a scan is waiting for right now; the workers take
+ // them before the decodes requested ahead of need, so a query's progress is not
+ // queued behind other queries' read-ahead.
+ urgent []*decodeTask
+ queue []*decodeTask
+ mu sync.Mutex
+ wg sync.WaitGroup
+ reserved atomic.Int64
+ head int
+ workers int
+ closed bool
+}
+
+// newDecodePool starts workers goroutines (at least one). pm bounds the node's
+// reservations; nil leaves them unbounded.
+func newDecodePool(workers int, pm protector.Memory) *decodePool {
+ workers = max(workers, 1)
+ p := &decodePool{pm: pm, workers: workers}
+ p.cond = sync.NewCond(&p.mu)
+ p.wg.Add(workers)
+ liveDecodeWorkers.Add(int64(workers))
+ for range workers {
+ go p.run()
+ }
+ return p
+}
+
+func (p *decodePool) run() {
+ defer p.wg.Done()
+ defer liveDecodeWorkers.Add(-1)
+ for {
+ p.mu.Lock()
+ for len(p.urgent) == 0 && p.head == len(p.queue) && !p.closed {
+ p.cond.Wait()
+ }
+ if p.closed {
+ p.mu.Unlock()
+ return
+ }
+ var t *decodeTask
+ if n := len(p.urgent); n > 0 {
+ t = p.urgent[0]
+ p.urgent[0] = nil
+ p.urgent = p.urgent[1:]
+ if len(p.urgent) == 0 {
+ p.urgent = nil
+ }
+ } else {
+ t = p.queue[p.head]
+ p.queue[p.head] = nil
+ p.head++
+ if p.head == len(p.queue) {
+ p.queue, p.head = p.queue[:0], 0
+ }
+ }
+ p.mu.Unlock()
+ t.run()
+ }
+}
+
+// submit queues t, ahead of the read-ahead tasks when urgent. On a closed pool the
+// task is canceled instead.
+func (p *decodePool) submit(t *decodeTask, urgent bool) {
+ p.mu.Lock()
+ if p.closed {
+ p.mu.Unlock()
+ t.cancel(errDecodePoolClosed)
+ return
+ }
+ if urgent {
+ p.urgent = append(p.urgent, t)
+ } else {
+ p.queue = append(p.queue, t)
+ }
+ p.mu.Unlock()
+ p.cond.Signal()
+}
+
+// close cancels the queued tasks and waits for the running ones, so no worker
+// touches a part once it returns. It is idempotent.
+func (p *decodePool) close() {
+ if p == nil {
+ return
+ }
+ p.mu.Lock()
+ if !p.closed {
+ p.closed = true
+ for _, t := range p.urgent {
+ t.cancel(errDecodePoolClosed)
+ }
+ for i := p.head; i < len(p.queue); i++ {
+ p.queue[i].cancel(errDecodePoolClosed)
+ p.queue[i] = nil
+ }
+ p.urgent, p.queue, p.head = nil, nil, 0
+ }
+ p.mu.Unlock()
+ p.cond.Broadcast()
+ p.wg.Wait()
+}
+
+// headroom is what the protector says is still free, or a negative number when it
+// sets no limit.
+func (p *decodePool) headroom() int64 {
+ if p.pm == nil {
+ return -1
+ }
+ return p.pm.AvailableBytes()
+}
+
+// reserve adds n to the node's total. A reservation made ahead of need (prefetch)
+// is refused when the total would pass the protector's headroom; the one a scan
+// needs to make progress is always added, and waits on the protector instead
+// (scanBudget.reserveCurrent), because refusing it could leave two queries each
+// waiting for memory the other holds.
+func (p *decodePool) reserve(n int64, ahead bool) bool {
+ if !ahead {
+ p.reserved.Add(n)
+ return true
+ }
+ free := p.headroom()
+ for {
+ cur := p.reserved.Load()
+ if free >= 0 && cur+n > free {
+ return false
+ }
+ if p.reserved.CompareAndSwap(cur, cur+n) {
+ return true
+ }
+ }
+}
+
+// scanReservation is the memory one cursor holds against its query's budget and the
+// node's total. It lives in the prefetcher's item for the cursor and is released
+// when the cursor's last reference goes.
+type scanReservation struct {
+ budget *scanBudget
+ bytes int64
+ released atomic.Bool
+}
+
+// release returns the bytes. It is idempotent and safe after the scan is closed.
+func (r *scanReservation) release() {
+ if r.budget == nil || !r.released.CompareAndSwap(false, true) {
+ return
+ }
+ r.budget.give(r.bytes)
+}
+
+type reserveOutcome uint8
+
+const (
+ reserveOK reserveOutcome = iota
+ // reserveOverQuery means the query's own budget is exhausted.
+ reserveOverQuery
+ // reserveOverNode means the node's reservations leave no room.
+ reserveOverNode
+)
+
+// scanBudget is one query's reservations. A query holds at most its quota at once;
+// the node-wide total additionally keeps concurrent queries within the
+// protector's headroom.
+type scanBudget struct {
+ tracker *vectorized.MemoryTracker
+ node *decodePool
+ held atomic.Int64
+ quota int64
+}
+
+// newScanBudget returns a budget of quota bytes (negative: no per-query limit)
+// that also counts against node's total (nil: no node accounting).
+func newScanBudget(quota int64, node *decodePool) *scanBudget {
+ b := &scanBudget{quota: quota, node: node}
+ if quota >= 0 {
+ b.tracker = vectorized.NewMemoryTracker(quota)
+ }
+ return b
+}
+
+// tryReserve reserves size bytes into r, reporting why it could not.
+func (b *scanBudget) tryReserve(size uint64, ahead bool, r *scanReservation) reserveOutcome {
+ n := int64(size)
+ if b.tracker != nil && b.tracker.Reserve(n) != nil {
+ return reserveOverQuery
+ }
+ if b.node != nil && !b.node.reserve(n, ahead) {
+ if b.tracker != nil {
+ b.tracker.Release(n)
+ }
+ return reserveOverNode
+ }
+ b.held.Add(n)
+ liveReservedBytes.Add(n)
+ r.budget, r.bytes = b, n
+ r.released.Store(false)
+ return reserveOK
+}
+
+func (b *scanBudget) give(n int64) {
+ b.held.Add(-n)
+ liveReservedBytes.Add(-n)
+ if b.tracker != nil {
+ b.tracker.Release(n)
+ }
+ if b.node != nil {
+ b.node.reserved.Add(-n)
+ }
+}
+
+// reserveCurrent reserves what the scan needs to make progress. It fails, with the
+// error the one-shot check used to return, only when the query alone would hold
+// more than its quota.
+//
+// Progress rule: the wait on the protector (AcquireResource, which holds a slot
+// while it backs off until process memory has room) is the admission of a query,
+// so it happens only while the query holds nothing, before its first reservation.
+// Once a query holds memory (a decoded frontier, borrowed batches, prefetched
+// cursors) the current cursor never blocks on the protector: two queries that each
+// waited for memory the other holds would stay stuck until their contexts end. The
+// per-query quota still bounds what a query holds, and the node's headroom gates
+// only the prefetch (decodePool.reserve).
+func (b *scanBudget) reserveCurrent(ctx context.Context, size uint64, r *scanReservation) error {
+ admit := b.held.Load() == 0
+ if b.tryReserve(size, false, r) == reserveOverQuery {
+ return fmt.Errorf("block scan quota exceeded: used %d bytes, quota is %d bytes", uint64(b.held.Load())+size, b.quota)
+ }
+ if admit && b.node != nil && b.node.pm != nil {
+ if err := b.node.pm.AcquireResource(ctx, size); err != nil {
+ r.release()
+ return err
+ }
+ }
+ return nil
+}
diff --git a/banyand/measure/scan_merge.go b/banyand/measure/scan_merge.go
new file mode 100644
index 000000000..b29459d5d
--- /dev/null
+++ b/banyand/measure/scan_merge.go
@@ -0,0 +1,842 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "cmp"
+ "context"
+ "slices"
+
+ "github.com/pkg/errors"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/bytes"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+// Time-ordered merge (measure scan path v2, step 4c).
+//
+// timeMerge replaces the legacy "decode every block, then merge row by row" path
+// for the blocks whose rows interleave in time: the series of a time-ordered query
+// whose time ranges overlap, and the overlapping blocks of one series. It works in
+// two steps per output batch.
+//
+// 1. Keys first. A heap over the head row of every live cursor yields rows in the
+// legacy order (queryResult.Less: timestamp, then series id, then the higher
+// version first) and applies the legacy duplicate rule (mergeBatch: of the rows
+// of one series sharing a timestamp, the highest version wins, the first one
+// popped on a tie). Only timestamp, version and series id are looked at; the
+// result is the batch's metadata columns and a winners list of (cursor, row).
+// 2. Gather. Each output column is built once from the winners, with a loop typed
+// by the column: numerics are an index copy, string, bytes and string-array
+// cells are views over the winning cursor's decoded bytes (the batch pins that
+// cursor), and only passthrough cells, index-substituted tags, missing columns
+// and cells whose stored type is not the projected one are decoded or filled
+// one cell at a time, exactly as the legacy copy does.
+//
+// Decoding is lazy. Cursors are ordered by the first timestamp the merge can read
+// from them (the last, going down, for a descending scan) and a cursor is decoded
+// only when the merge frontier reaches that timestamp, so the decoded memory is
+// that of the blocks overlapping the current instant, not of the whole result. A
+// cursor is released as soon as the merge has consumed its last row; batches that
+// borrow from it keep it alive through their own references.
+//
+// cursorLoader is how a merge asks for the next cursor: the scan hands it the
+// prefetcher's take, so the cursors the merge reaches have usually been decoded by
+// the node's decode pool already (scan_prefetch.go).
+
+// mergeFirstPageRows and mergeMaxPageRows bound the rows of one merged batch. The
+// first page is small so a query with a limit does not gather rows it never reads;
+// the page then doubles up to a block's worth of rows (8,192), the size the
+// zero-copy path already hands the pipeline, which amortizes the per-page work
+// (the column plan, owner pins) without holding more than a block of rows in
+// flight. The legacy merge cut pages at 4,096 rows.
+const (
+ minFinishedLimit = 8
+ mergeFirstPageRows = 1024
+ mergeMaxPageRows = 8192
+)
+
+// cursorLoader decodes bc, a cursor planned from block metadata only, for the
+// scan: it reports whether the block has rows in the query time range. On true the
+// cursor is positioned at its first row for the scan direction (the last row for
+// a descending scan). The caller releases the cursor when it reports false or an
+// error.
+type cursorLoader func(ctx context.Context, bc *blockCursor, desc bool) (bool, error)
+
+// decodeCursor is the default cursorLoader: it decodes the block on the calling
+// goroutine.
+func decodeCursor(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ select {
+ case <-ctx.Done():
+ return false, ctx.Err()
+ default:
+ }
+ tmpBlock := generateBlock()
+ ok := bc.loadData(tmpBlock, true)
+ releaseBlock(tmpBlock)
+ if !ok {
+ return false, nil
+ }
+ if desc {
+ bc.idx = len(bc.timestamps) - 1
+ }
+ return true, nil
+}
+
+// srcKind says how one projected column is read from one cursor.
+type srcKind uint8
+
+const (
+ // srcOther is decoded cell by cell through the legacy helpers.
+ srcOther srcKind = iota
+ // srcNull is a column the cursor lacks, or whose stored type is not the schema's.
+ srcNull
+ // srcHidden is a tag substituted from the index.
+ srcHidden
+ // srcInt64 and srcFloat64 are typed numeric columns read by index.
+ srcInt64
+ srcFloat64
+ // srcStr, srcBin and srcStrArr are bytes-form columns lent as views.
+ srcStr
+ srcBin
+ srcStrArr
+)
+
+// colSource is how one cursor serves one projected column; it is resolved once,
+// when the cursor is decoded.
+type colSource struct {
+ col *column
+ hidden *modelv1.TagValue
+ kind srcKind
+}
+
+// colPlan is one projected tag or field column of the output.
+type colPlan struct {
+ def vectorized.ColumnDef
+ out int // index into MeasureBatch.Tags or Fields
+ ord int // index into mergeEntry.src
+ isTag bool
+}
+
+func newColPlan(schema *vectorized.BatchSchema) []colPlan {
+ var plan []colPlan
+ tagIdx, fieldIdx := 0, 0
+ for _, def := range schema.Columns {
+ switch def.Role {
+ case vectorized.RoleTag:
+ plan = append(plan, colPlan{def: def, out: tagIdx, ord: len(plan), isTag: true})
+ tagIdx++
+ case vectorized.RoleField:
+ plan = append(plan, colPlan{def: def, out: fieldIdx, ord: len(plan)})
+ fieldIdx++
+ case vectorized.RoleTimestamp, vectorized.RoleVersion,
+ vectorized.RoleSeriesID, vectorized.RoleShardID,
+ vectorized.RoleElementID, vectorized.RoleOrderKey:
+ }
+ }
+ return plan
+}
+
+// resolveSources decides, for every planned column, how bc serves it. The order
+// of the checks is the legacy order (fillTagColumnAllRows, fillFieldColumnAllRows).
+func resolveSources(bc *blockCursor, plan []colPlan, indexValue map[string]*modelv1.TagValue, dst []colSource) []colSource {
+ dst = dst[:0]
+ for i := range plan {
+ cp := &plan[i]
+ if cp.isTag {
+ dst = append(dst, resolveTagSource(bc, cp.def, indexValue))
+ } else {
+ dst = append(dst, resolveFieldSource(bc, cp.def))
+ }
+ }
+ return dst
+}
+
+func resolveTagSource(bc *blockCursor, def vectorized.ColumnDef, indexValue map[string]*modelv1.TagValue) colSource {
+ if indexValue != nil && indexValue[def.Name] != nil {
+ return colSource{kind: srcHidden, hidden: indexValue[def.Name]}
+ }
+ cf := bc.findTagFamily(def.TagFamily)
+ if cf == nil {
+ return colSource{kind: srcNull}
+ }
+ column := cf.findColumn(def.Name)
+ if column == nil {
+ return colSource{kind: srcNull}
+ }
+ schemaType, hasSchemaType := bc.schemaTagTypes[def.Name]
+ if !hasSchemaType || column.valueType != schemaType {
+ return colSource{kind: srcNull}
+ }
+ src := colSource{kind: srcOther, col: column}
+ switch column.form {
+ case numericFormInt64:
+ if def.Type == vectorized.ColumnTypeInt64 {
+ src.kind = srcInt64
+ }
+ case numericFormBytes:
+ switch {
+ case def.Type == vectorized.ColumnTypeString && column.valueType == pbv1.ValueTypeStr:
+ src.kind = srcStr
+ case def.Type == vectorized.ColumnTypeBytes && column.valueType == pbv1.ValueTypeBinaryData:
+ src.kind = srcBin
+ case def.Type == vectorized.ColumnTypeStrArray && column.valueType == pbv1.ValueTypeStrArr:
+ src.kind = srcStrArr
+ }
+ case numericFormFloat64:
+ }
+ return src
+}
+
+func resolveFieldSource(bc *blockCursor, def vectorized.ColumnDef) colSource {
+ column := bc.findFieldColumn(def.Name)
+ if column == nil {
+ return colSource{kind: srcNull}
+ }
+ src := colSource{kind: srcOther, col: column}
+ switch column.form {
+ case numericFormInt64:
+ if def.Type == vectorized.ColumnTypeInt64 {
+ src.kind = srcInt64
+ }
+ case numericFormFloat64:
+ if def.Type == vectorized.ColumnTypeFloat64 {
+ src.kind = srcFloat64
+ }
+ case numericFormBytes:
+ }
+ return src
+}
+
+// mergeEntry is a decoded cursor in the merge, with the key of its head row cached
+// so the heap never reads the cursor's slices.
+type mergeEntry struct {
+ bc *blockCursor
+ src []colSource
+ // ts, ver and sid are the key of the row at idx.
+ ts int64
+ ver int64
+ sid common.SeriesID
+ idx int
+ // seq is the cursor's position in the planned order; it breaks ties between
+ // rows whose keys are equal.
+ seq int
+ // pageStamp and pageSlot say whether, and where, the entry is in the current
+ // page's table.
+ pageStamp uint32
+ pageSlot int
+}
+
+// mergeWinner is a row that survives into the output: the entry's slot in the
+// page table and its row in the cursor.
+type mergeWinner struct {
+ slot uint32
+ row uint32
+}
+
+// pageEntry is an entry as the page uses it.
+type pageEntry struct {
+ e *mergeEntry
+ pinned bool
+}
+
+// pendingCursor is a cursor not yet decoded and the first timestamp the merge can
+// read from it.
+type pendingCursor struct {
+ bc *blockCursor
+ start int64
+}
+
+// timeMerge merges the rows of a set of cursors into batches in time order. A
+// merge is recycled for the next one through newTimeMerge: the scan runs one merge at a
+// time, so the buffers it grew are reused instead of reallocated per series.
+type timeMerge struct {
+ ctx context.Context
+ qr *queryResult
+ stats *scanStats
+ load cursorLoader
+ pf *prefetcher
+ schema *vectorized.BatchSchema
+ indexValues map[common.SeriesID]map[string]*modelv1.TagValue
+ // pending are the cursors not yet decoded, in the order the frontier reaches
+ // them.
+ pending []pendingCursor
+ heap []*mergeEntry
+ // free holds entries of consumed cursors, ready for the next decode.
+ free []*mergeEntry
+ finished []*mergeEntry
+ pageEntries []pageEntry
+ winners []mergeWinner
+ plan []colPlan
+ i64Scratch [][]int64
+ f64Scratch [][]float64
+ boundaries []int
+ rows int64
+ cursors int64
+ pi int
+ live int
+ peakLive int
+ frontier int
+ pageRows int
+ page uint32
+ desc bool
+ closed bool
+}
+
+// newTimeMerge plans a merge over cursors, which it takes ownership of. The
+// cursors are undecoded; their metadata orders them. plan is the output column
+// plan of the query's schema (newColPlan), shared by every merge of a scan; spare,
+// when not nil, is a finished merge whose buffers the new one reuses.
+func newTimeMerge(qr *queryResult, stats *scanStats, load cursorLoader, plan []colPlan, cursors []*blockCursor,
+ desc bool, spare *timeMerge,
+) *timeMerge {
+ m := spare
+ if m == nil {
+ m = &timeMerge{}
+ }
+ m.ctx, m.qr, m.stats, m.load, m.pf = qr.ctx, qr, stats, load, nil
+ m.schema, m.indexValues, m.plan, m.desc = qr.batchSchema, qr.storedIndexValue, plan, desc
+ m.pageRows, m.pi, m.live, m.peakLive, m.frontier = mergeFirstPageRows, 0, 0, 0, 0
+ m.rows, m.cursors, m.page, m.closed = 0, 0, 0, false
+ m.pending = m.pending[:0]
+ for _, bc := range cursors {
+ m.pending = append(m.pending, pendingCursor{bc: bc, start: mergeStart(bc, desc)})
+ }
+ // Equal starts keep the order the cursors arrived in.
+ slices.SortStableFunc(m.pending, func(a, b pendingCursor) int {
+ if desc {
+ return cmp.Compare(b.start, a.start)
+ }
+ return cmp.Compare(a.start, b.start)
+ })
+ m.stats.mergeUnits++
+ return m
+}
+
+// mergeStart is the first timestamp the merge can read from bc: its first (last,
+// descending) timestamp clamped to the query time range.
+func mergeStart(bc *blockCursor, desc bool) int64 {
+ if desc {
+ return min(bc.bm.timestamps.max, bc.maxTimestamp)
+ }
+ return max(bc.bm.timestamps.min, bc.minTimestamp)
+}
+
+// less is the legacy queryResult.Less for a time-ordered query, with the position
+// in the planned order as the last tie-break.
+func (m *timeMerge) less(a, b *mergeEntry) bool {
+ if a.ts != b.ts {
+ if m.desc {
+ return a.ts > b.ts
+ }
+ return a.ts < b.ts
+ }
+ if a.sid != b.sid {
+ return a.sid < b.sid
+ }
+ if a.ver != b.ver {
+ return a.ver > b.ver
+ }
+ return a.seq < b.seq
+}
+
+func (m *timeMerge) siftUp(i int) {
+ h := m.heap
+ for i > 0 {
+ parent := (i - 1) / 2
+ if !m.less(h[i], h[parent]) {
+ return
+ }
+ h[i], h[parent] = h[parent], h[i]
+ i = parent
+ }
+}
+
+func (m *timeMerge) siftDown(i int) {
+ h := m.heap
+ n := len(h)
+ for {
+ child := 2*i + 1
+ if child >= n {
+ return
+ }
+ if right := child + 1; right < n && m.less(h[right], h[child]) {
+ child = right
+ }
+ if !m.less(h[child], h[i]) {
+ return
+ }
+ h[i], h[child] = h[child], h[i]
+ i = child
+ }
+}
+
+// fixTop restores the heap after the key of its root grew. A cursor that keeps
+// winning (one series, or a run inside a block) stays at the root after two
+// comparisons. Otherwise the new key is usually larger than most of the heap (the
+// series that just emitted a timestamp has the largest key among those sharing
+// it), so instead of comparing against both children on the way down, the hole is
+// walked down the smaller children to a leaf, one comparison per level, and the
+// entry climbs back up from there.
+func (m *timeMerge) fixTop() {
+ h := m.heap
+ n := len(h)
+ e := h[0]
+ if n < 3 {
+ m.siftDown(0)
+ return
+ }
+ if !m.less(h[1], e) && !m.less(h[2], e) {
+ return
+ }
+ i := 0
+ for {
+ child := 2*i + 1
+ if child >= n {
+ break
+ }
+ if right := child + 1; right < n && m.less(h[right], h[child]) {
+ child = right
+ }
+ h[i] = h[child]
+ i = child
+ }
+ for i > 0 {
+ parent := (i - 1) / 2
+ if !m.less(e, h[parent]) {
+ break
+ }
+ h[i] = h[parent]
+ i = parent
+ }
+ h[i] = e
+}
+
+// reaches reports whether the frontier, the head row of the heap, has reached the
+// first timestamp start can be read from, so the cursor must join the merge. An
+// equal timestamp counts: the cursor may hold a row that sorts before the head's.
+func (m *timeMerge) reaches(start, head int64) bool {
+ if m.desc {
+ return start >= head
+ }
+ return start <= head
+}
+
+// fill decodes every pending cursor the frontier has reached. With an empty heap
+// it decodes the next pending cursor, whatever its start.
+func (m *timeMerge) fill() error {
+ for m.pi < len(m.pending) {
+ if len(m.heap) > 0 && !m.reaches(m.pending[m.pi].start, m.heap[0].ts) {
+ return nil
+ }
+ bc := m.pending[m.pi].bc
+ seq := m.pi
+ m.pending[m.pi].bc = nil
+ m.pi++
+ ok, err := m.load(m.ctx, bc, m.desc)
+ if err != nil {
+ releaseBlockCursor(bc)
+ return errors.WithMessagef(err, "interrupt: hit %d", m.qr.hit)
+ }
+ if !ok {
+ releaseBlockCursor(bc)
+ continue
+ }
+ m.stats.mergeCursors++
+ m.cursors++
+ m.live++
+ m.peakLive = max(m.peakLive, m.live)
+ var e *mergeEntry
+ if n := len(m.free); n > 0 {
+ e = m.free[n-1]
+ m.free[n-1] = nil
+ m.free = m.free[:n-1]
+ } else {
+ e = &mergeEntry{}
+ }
+ e.bc, e.seq, e.idx, e.sid, e.pageStamp = bc, seq, bc.idx, bc.bm.seriesID, 0
+ var indexValue map[string]*modelv1.TagValue
+ if m.indexValues != nil {
+ indexValue = m.indexValues[e.sid]
+ }
+ e.src = resolveSources(bc, m.plan, indexValue, e.src)
+ e.ts, e.ver = bc.timestamps[e.idx], bc.versions[e.idx]
+ m.heap = append(m.heap, e)
+ m.siftUp(len(m.heap) - 1)
+ m.frontier = max(m.frontier, len(m.heap))
+ m.pf.setFrontier(m.frontier)
+ }
+ return nil
+}
+
+// nextPage returns the next merged batch, or nil when every cursor is consumed.
+func (m *timeMerge) nextPage(ctx context.Context) (*model.MeasureBatch, error) {
+ select {
+ case <-ctx.Done():
+ return nil, errors.WithMessagef(ctx.Err(), "interrupt: hit %d", m.qr.hit)
+ default:
+ }
+ m.page++
+ m.winners = m.winners[:0]
+ m.pageEntries = m.pageEntries[:0]
+ b := newMeasureBatchForSchema(m.schema, m.pageRows)
+ if err := m.mergeKeys(b); err != nil {
+ b.Release()
+ m.releaseFinished()
+ return nil, err
+ }
+ if len(m.winners) == 0 {
+ b.Release()
+ return nil, nil
+ }
+ m.gather(b)
+ m.markSeriesRuns(b)
+ m.releaseFinished()
+ m.stats.mergePages++
+ m.stats.mergeRows += int64(len(m.winners))
+ m.rows += int64(len(m.winners))
+ m.pageRows = min(m.pageRows*2, mergeMaxPageRows)
+ return b, nil
+}
+
+// finishedLimit is how many consumed-but-unreleased cursors a page may accumulate:
+// they stay decoded until the page is gathered, and the cursors that replace them
+// are decoded in the meantime, so an unbounded page that crosses a block boundary
+// of every series would hold twice the frontier. A quarter of the widest frontier
+// seen keeps the peak within a quarter of one decoded block per series; the floor
+// keeps small merges (a series' two or three overlapping blocks) in one page.
+func (m *timeMerge) finishedLimit() int {
+ return max(m.frontier/4, minFinishedLimit)
+}
+
+// mergeKeys pops rows in order into b's metadata columns and the winners list.
+//
+// The duplicate rule is mergeBatch's: a row with the same timestamp and series as
+// the last winner replaces it when its version is higher, and is dropped
+// otherwise. Unlike mergeBatch, a full page is not closed in the middle of such a
+// group (mergeBatch closed at 4,096 rows wherever it stood, so a duplicate that
+// straddled the cut was emitted as a second row), which keeps the dedup exact
+// whatever the page size.
+func (m *timeMerge) mergeKeys(b *model.MeasureBatch) error {
+ step := 1
+ if m.desc {
+ step = -1
+ }
+ var lastTS, lastVer int64
+ var lastSID common.SeriesID
+ for {
+ if err := m.fill(); err != nil {
+ return err
+ }
+ if len(m.heap) == 0 {
+ return nil
+ }
+ top := m.heap[0]
+ n := len(m.winners)
+ duplicate := n > 0 && top.ts == lastTS && top.sid == lastSID
+ if (n >= m.pageRows || len(m.finished) >= m.finishedLimit()) && !duplicate {
+ return nil
+ }
+ switch {
+ case !duplicate:
+ m.appendWinner(b, top)
+ lastTS, lastVer, lastSID = top.ts, top.ver, top.sid
+ case top.ver > lastVer:
+ m.replaceWinner(b, top)
+ lastVer = top.ver
+ }
+ top.idx += step
+ if top.idx < 0 || top.idx >= len(top.bc.timestamps) {
+ last := len(m.heap) - 1
+ m.heap[0] = m.heap[last]
+ m.heap[last] = nil
+ m.heap = m.heap[:last]
+ if last > 0 {
+ m.siftDown(0)
+ }
+ m.finished = append(m.finished, top)
+ continue
+ }
+ top.ts, top.ver = top.bc.timestamps[top.idx], top.bc.versions[top.idx]
+ m.fixTop()
+ }
+}
+
+func (m *timeMerge) slotOf(e *mergeEntry) uint32 {
+ if e.pageStamp != m.page {
+ e.pageStamp = m.page
+ e.pageSlot = len(m.pageEntries)
+ m.pageEntries = append(m.pageEntries, pageEntry{e: e})
+ }
+ return uint32(e.pageSlot)
+}
+
+func (m *timeMerge) appendWinner(b *model.MeasureBatch, e *mergeEntry) {
+ b.Timestamps = append(b.Timestamps, e.ts)
+ b.Versions = append(b.Versions, e.ver)
+ b.ShardIDs = append(b.ShardIDs, e.bc.shardID)
+ b.SeriesIDs = append(b.SeriesIDs, e.sid)
+ m.winners = append(m.winners, mergeWinner{slot: m.slotOf(e), row: uint32(e.idx)})
+}
+
+func (m *timeMerge) replaceWinner(b *model.MeasureBatch, e *mergeEntry) {
+ last := len(m.winners) - 1
+ b.Versions[last] = e.ver
+ b.ShardIDs[last] = e.bc.shardID
+ m.winners[last] = mergeWinner{slot: m.slotOf(e), row: uint32(e.idx)}
+}
+
+// releaseFinished drops the merge's reference on every cursor whose rows are all
+// consumed. It runs after the gather, which reads the cursors of the page's
+// winners; a batch that borrows from a cursor holds its own reference.
+func (m *timeMerge) releaseFinished() {
+ for i, e := range m.finished {
+ releaseBlockCursor(e.bc)
+ e.bc = nil
+ clear(e.src)
+ m.free = append(m.free, e)
+ m.finished[i] = nil
+ m.live--
+ }
+ m.finished = m.finished[:0]
+}
+
+// markSeriesRuns records, when the batch interleaves several series, where each run
+// of one series ends (MeasureBatch.SeriesBoundaries).
+func (m *timeMerge) markSeriesRuns(b *model.MeasureBatch) {
+ ids := b.SeriesIDs
+ m.boundaries = m.boundaries[:0]
+ for i := 1; i < len(ids); i++ {
+ if ids[i] != ids[i-1] {
+ m.boundaries = append(m.boundaries, i)
+ }
+ }
+ if len(m.boundaries) == 0 {
+ return
+ }
+ b.SeriesBoundaries = append(b.SeriesBoundaries[:0], m.boundaries...)
+ b.SeriesBoundaries = append(b.SeriesBoundaries, len(ids))
+}
+
+// gather builds every projected column of b from the winners.
+func (m *timeMerge) gather(b *model.MeasureBatch) {
+ for i := range m.plan {
+ cp := &m.plan[i]
+ var col vectorized.Column
+ if cp.isTag {
+ col = b.Tags[cp.out]
+ } else {
+ col = b.Fields[cp.out]
+ }
+ switch cp.def.Type {
+ case vectorized.ColumnTypeInt64:
+ if c, ok := col.(*vectorized.TypedColumn[int64]); ok {
+ gatherNumeric(m, c, cp, srcInt64, func(s *colSource) []int64 { return s.col.int64s }, &m.i64Scratch)
+ continue
+ }
+ case vectorized.ColumnTypeFloat64:
+ if c, ok := col.(*vectorized.TypedColumn[float64]); ok {
+ gatherNumeric(m, c, cp, srcFloat64, func(s *colSource) []float64 { return s.col.float64s }, &m.f64Scratch)
+ continue
+ }
+ case vectorized.ColumnTypeString:
+ if c, ok := col.(*vectorized.TypedColumn[string]); ok {
+ m.gatherStrings(b, c, cp)
+ continue
+ }
+ case vectorized.ColumnTypeBytes:
+ if c, ok := col.(*vectorized.TypedColumn[[]byte]); ok {
+ m.gatherBytes(b, c, cp)
+ continue
+ }
+ case vectorized.ColumnTypeStrArray:
+ if c, ok := col.(*vectorized.TypedColumn[[]string]); ok {
+ m.gatherStrArrays(b, c, cp)
+ continue
+ }
+ case vectorized.ColumnTypeInt64Array, vectorized.ColumnTypeTagValue, vectorized.ColumnTypeFieldValue:
+ }
+ m.gatherCells(col, cp)
+ }
+}
+
+// cell appends the row of winner w when its source is not served by a typed loop:
+// a null, an index-substituted tag, or a cell decoded the legacy way.
+func (m *timeMerge) cell(col vectorized.Column, cp *colPlan, s *colSource, row int) {
+ switch s.kind {
+ case srcNull:
+ if cp.isTag {
+ appendNullTagN(col, cp.def, 1)
+ } else {
+ appendNullFieldN(col, cp.def, 1)
+ }
+ case srcHidden:
+ appendTagValueAt(col, cp.def, s.hidden)
+ case srcOther, srcInt64, srcFloat64, srcStr, srcBin, srcStrArr:
+ if cp.isTag {
+ fillTagRows(col, cp.def, s.col, row, row+1, false)
+ } else {
+ fillFieldRows(col, cp.def, s.col, row, row+1, false)
+ }
+ }
+}
+
+// gatherCells builds a column whose cells all go through the legacy per-cell decode:
+// passthrough columns and array types that do not borrow.
+func (m *timeMerge) gatherCells(col vectorized.Column, cp *colPlan) {
+ for _, w := range m.winners {
+ e := m.pageEntries[w.slot].e
+ m.cell(col, cp, &e.src[cp.ord], int(w.row))
+ }
+}
+
+// gatherNumeric builds a typed numeric column. When every cursor of the page
+// serves it as a typed slice the loop is a plain index copy.
+func gatherNumeric[T int64 | float64](m *timeMerge, c *vectorized.TypedColumn[T], cp *colPlan, kind srcKind,
+ data func(*colSource) []T, scratch *[][]T,
+) {
+ slices := (*scratch)[:0]
+ allTyped := true
+ for i := range m.pageEntries {
+ s := &m.pageEntries[i].e.src[cp.ord]
+ if s.kind != kind {
+ allTyped = false
+ break
+ }
+ slices = append(slices, data(s))
+ }
+ *scratch = slices
+ if allTyped {
+ for _, w := range m.winners {
+ c.Append(slices[w.slot][w.row])
+ }
+ return
+ }
+ for _, w := range m.winners {
+ s := &m.pageEntries[w.slot].e.src[cp.ord]
+ if s.kind == kind {
+ c.Append(data(s)[w.row])
+ continue
+ }
+ m.cell(c, cp, s, int(w.row))
+ }
+}
+
+// pin makes the batch hold the cursor of page slot, once per page.
+func (m *timeMerge) pin(b *model.MeasureBatch, slot uint32) {
+ pe := &m.pageEntries[slot]
+ if !pe.pinned {
+ pe.pinned = true
+ b.Owners.Pin(pe.e.bc)
+ }
+}
+
+func (m *timeMerge) gatherStrings(b *model.MeasureBatch, c *vectorized.TypedColumn[string], cp *colPlan) {
+ for _, w := range m.winners {
+ s := &m.pageEntries[w.slot].e.src[cp.ord]
+ if s.kind != srcStr {
+ m.cell(c, cp, s, int(w.row))
+ continue
+ }
+ raw := s.col.values[w.row]
+ if raw == nil {
+ c.AppendNull()
+ continue
+ }
+ if len(raw) > 0 {
+ m.pin(b, w.slot)
+ }
+ c.Append(convert.BytesToString(raw))
+ }
+}
+
+func (m *timeMerge) gatherBytes(b *model.MeasureBatch, c *vectorized.TypedColumn[[]byte], cp *colPlan) {
+ for _, w := range m.winners {
+ s := &m.pageEntries[w.slot].e.src[cp.ord]
+ if s.kind != srcBin {
+ m.cell(c, cp, s, int(w.row))
+ continue
+ }
+ raw := s.col.values[w.row]
+ if raw == nil {
+ c.AppendNull()
+ continue
+ }
+ if len(raw) > 0 {
+ m.pin(b, w.slot)
+ }
+ c.Append(raw[:len(raw):len(raw)])
+ }
+}
+
+func (m *timeMerge) gatherStrArrays(b *model.MeasureBatch, c *vectorized.TypedColumn[[]string], cp *colPlan) {
+ var bb *bytes.Buffer
+ var slab []string
+ defer func() {
+ if bb != nil {
+ bigValuePool.Release(bb)
+ }
+ }()
+ for _, w := range m.winners {
+ s := &m.pageEntries[w.slot].e.src[cp.ord]
+ if s.kind != srcStrArr {
+ m.cell(c, cp, s, int(w.row))
+ continue
+ }
+ if bb == nil {
+ bb = bigValuePool.Generate()
+ }
+ if appendBorrowedStrArr(c, s.col.values[w.row], &slab, bb) {
+ m.pin(b, w.slot)
+ }
+ }
+}
+
+// close releases every cursor the merge still holds. It is idempotent.
+func (m *timeMerge) close() {
+ if m.closed {
+ return
+ }
+ m.closed = true
+ for i, e := range m.heap {
+ releaseBlockCursor(e.bc)
+ e.bc = nil
+ m.heap[i] = nil
+ }
+ m.heap = m.heap[:0]
+ m.releaseFinished()
+ for i := m.pi; i < len(m.pending); i++ {
+ if bc := m.pending[i].bc; bc != nil {
+ releaseBlockCursor(bc)
+ m.pending[i].bc = nil
+ }
+ }
+ m.pending = m.pending[:0]
+ m.live = 0
+ m.stats.mergePeakLive = max(m.stats.mergePeakLive, int64(m.peakLive))
+ if qm := m.qr.qm; qm != nil {
+ qm.scanMergeRows.Inc(float64(m.rows))
+ qm.scanMergeCursors.Inc(float64(m.cursors))
+ qm.scanMergePeakLive.Observe(float64(m.peakLive))
+ }
+}
diff --git a/banyand/measure/scan_merge_test.go b/banyand/measure/scan_merge_test.go
new file mode 100644
index 000000000..e8c3a2ded
--- /dev/null
+++ b/banyand/measure/scan_merge_test.go
@@ -0,0 +1,688 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "container/heap"
+ "context"
+ "fmt"
+ "math/rand"
+ "slices"
+ "sort"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+)
+
+// oapFixture is the shape OAP produces: many series that all cover the same time
+// range, flushed in several parts that overlap, repeat timestamps with a higher
+// version, or only touch each other.
+//
+// P1: every series, 1..300, v1
+// P2: even series, 250..500, v2 (overlaps P1's tail, rewrites 250..300)
+// P3: every fifth series, 100..260, v3 (overlaps both, rewrites 250..260 again)
+// P4: every series, 501..650, v4 (disjoint from everything before it)
+// P5: the first ten series, 1..50, v1 again (same version as P1: identical rows)
+func oapFixture(t testing.TB, series int) *fxFixture {
+ var p1, p2, p3, p4, p5 fxPart
+ for sid := common.SeriesID(1); sid <= common.SeriesID(series); sid++ {
+ p1 = append(p1, fxRange{sid, 1, 300, 1})
+ if sid%2 == 0 {
+ p2 = append(p2, fxRange{sid, 250, 500, 2})
+ }
+ if sid%5 == 0 {
+ p3 = append(p3, fxRange{sid, 100, 260, 3})
+ }
+ p4 = append(p4, fxRange{sid, 501, 650, 4})
+ if sid <= 10 {
+ p5 = append(p5, fxRange{sid, 1, 50, 1})
+ }
+ }
+ return newFixture(t, p1, p2, p3, p4, p5)
+}
+
+// staggeredFixture has series whose ranges start at different times and overlap
+// partially, with duplicate timestamps across parts and blocks that touch at one
+// timestamp.
+func staggeredFixture(t testing.TB, series int) *fxFixture {
+ var a, b, c fxPart
+ for i := 1; i <= series; i++ {
+ sid := common.SeriesID(i)
+ start := int64(i * 20)
+ a = append(a, fxRange{sid, start, start + 300, 1})
+ if i%3 != 0 {
+ b = append(b, fxRange{sid, start + 150, start + 450, 2})
+ }
+ if i%4 == 0 {
+ c = append(c, fxRange{sid, start + 450, start + 600, 3})
+ }
+ }
+ return newFixture(t, a, b, c)
+}
+
+func hiddenTags() map[common.SeriesID]map[string]*modelv1.TagValue {
+ return map[common.SeriesID]map[string]*modelv1.TagValue{
+ 2: {fxTagHid: strTagValue("hidden-2")},
+ 7: {fxTagHid: strTagValue("hidden-7")},
+ 10: {fxTagHid: strTagValue("hidden-10")},
+ }
+}
+
+func TestTimeMergeMatchesLegacyAcrossGeneratedQueries(t *testing.T) {
+ verifyNoScanLeaks(t)
+ oapSeries, staggeredSeries := 100, 24
+ if raceEnabled {
+ oapSeries, staggeredSeries = 20, 12
+ }
+ fxNames := []string{"mixed", "oap", "staggered"}
+ fixtures := map[string]*fxFixture{
+ "oap": oapFixture(t, oapSeries),
+ "staggered": staggeredFixture(t, staggeredSeries),
+ "mixed": mixedFixture(t),
+ }
+ projections := []struct {
+ name string
+ tags []string
+ fields []string
+ }{
+ {"all", allTags, allFields},
+ {"numeric", []string{fxTagN2, fxTagN}, []string{fxFldInt, fxFldFlt, fxFldNul}},
+ {"strings and arrays", []string{fxTagSvc, fxTagURL, fxTagArr, fxTagIAr, fxTagBin}, []string{fxFldStr, fxFldBin}},
+ {"fields only", nil, []string{fxFldInt}},
+ {"tags only", []string{fxTagURL, fxTagHid}, nil},
+ }
+ rng := rand.New(rand.NewSource(20261014)) //nolint:gosec // a fixed seed makes the generated queries reproducible
+ ranges := [][2]int64{{0, 1 << 20}, {100, 450}, {299, 301}, {250, 260}, {501, 505}, {40, 40}}
+ limits := [][2]uint32{{0, 0}, {0, 7}, {5, 50}, {300, 2000}, {0, 100000}}
+ orders := []struct {
+ name string
+ orderByTS bool
+ ascTS bool
+ }{
+ {"no order by", true, true},
+ {"order by time asc", true, true},
+ {"order by time desc", true, false},
+ {"series order", false, true},
+ {"series order desc flag", false, false},
+ }
+ iters := 36
+ // Under the race detector the random picks are replaced by a pairwise cover of the
+ // order, projection, range and limit values (every value, and every pair of order,
+ // projection and limit), spread over the fixtures.
+ var cover [][]int
+ if raceEnabled {
+ var combos [][]int
+ for o := range orders {
+ for p := range projections {
+ for r := range ranges {
+ for l := range limits {
+ combos = append(combos, []int{o, p, r, l})
+ }
+ }
+ }
+ }
+ span := func(c []int) int { return int(min(ranges[c[2]][1]-ranges[c[2]][0], 700)) }
+ rows := func(c []int) int {
+ if lim := limits[c[3]]; lim[1] < 1000 {
+ return int(lim[1]) + 1
+ }
+ return 2000
+ }
+ cost := func(c []int) int { return span(c) * rows(c) / 100 }
+ cover = coverPairs(combos, [][2]int{{0, 1}, {0, 3}, {1, 3}}, cost)
+ // The mixed fixture is the slowest per row: it takes the cheapest queries and the
+ // small fixtures take the heaviest, so every fixture still sees every shape of query.
+ sort.SliceStable(cover, func(i, j int) bool { return cost(cover[i]) < cost(cover[j]) })
+ iters = (len(cover) + len(fxNames) - 1) / len(fxNames)
+ }
+ for fxIdx, name := range fxNames {
+ fx := fixtures[name]
+ for iter := range iters {
+ ord := orders[rng.Intn(len(orders))]
+ proj := projections[rng.Intn(len(projections))]
+ tr := ranges[rng.Intn(len(ranges))]
+ lim := limits[rng.Intn(len(limits))]
+ if raceEnabled {
+ idx := fxIdx*iters + iter
+ if idx >= len(cover) {
+ break
+ }
+ ord, proj, tr, lim = orders[cover[idx][0]], projections[cover[idx][1]], ranges[cover[idx][2]], limits[cover[idx][3]]
+ }
+ q := &fxQuery{
+ tags: proj.tags, fields: proj.fields, minTS: tr[0], maxTS: tr[1],
+ offset: lim[0], limit: lim[1], orderByTS: ord.orderByTS, ascTS: ord.ascTS, passthrough: rng.Intn(4) == 0,
+ }
+ if rng.Intn(3) != 0 {
+ order := slices.Clone(fx.sids)
+ rng.Shuffle(len(order), func(i, j int) { order[i], order[j] = order[j], order[i] })
+ q.order = order[:max(2, rng.Intn(len(order)+1))]
+ }
+ if rng.Intn(2) == 0 && len(proj.tags) > 0 {
+ q.stored = hiddenTags()
+ }
+ what := fmt.Sprintf("%s iter %d %s proj=%s range=%v limit=%v series=%d hidden=%v passthrough=%v",
+ name, iter, ord.name, proj.name, tr, lim, len(q.order), q.stored != nil, q.passthrough)
+ t.Run(fmt.Sprintf("%s/%d", name, iter), func(t *testing.T) {
+ want := fx.run(t, q, true)
+ got := fx.run(t, q, false)
+ requireSameResult(t, want, got, what)
+ if len(want.points) > 0 {
+ require.NotNil(t, got.scan, what)
+ }
+ })
+ }
+ }
+}
+
+func TestTimeMergeMatchesLegacyThroughGroupByAggregationAndTimeBuckets(t *testing.T) {
+ verifyNoScanLeaks(t)
+ oapSeries, staggeredSeries := 60, 16
+ if raceEnabled {
+ oapSeries, staggeredSeries = 12, 8
+ }
+ fxNames := []string{"oap", "staggered"}
+ fixtures := map[string]*fxFixture{"oap": oapFixture(t, oapSeries), "staggered": staggeredFixture(t, staggeredSeries)}
+ groupBy := &model.MeasureGroupBy{TagFamily: fxFamily, TagNames: []string{fxTagSvc}}
+ bucketOnly := &model.MeasureGroupBy{TimeBucket: &model.MeasureTimeBucket{Width: "100ns", WidthNanos: 100}}
+ bucketAndTag := &model.MeasureGroupBy{
+ TagFamily: fxFamily, TagNames: []string{fxTagSvc}, TimeBucket: &model.MeasureTimeBucket{Width: "64ns", WidthNanos: 64},
+ }
+ aggs := []*model.MeasureAgg{
+ {FieldName: fxFldInt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_SUM},
+ {FieldName: fxFldFlt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_MEAN},
+ {FieldName: fxFldInt, Func: modelv1.AggregationFunction_AGGREGATION_FUNCTION_MAX},
+ }
+ groupBys := []*model.MeasureGroupBy{nil, groupBy, bucketOnly, bucketAndTag}
+ ranges := [][2]int64{{0, 1 << 20}, {120, 480}}
+ // combos are [fixture, grouping, aggregation, range] indexes of the valid cross product.
+ var combos [][]int
+ for f := range fxNames {
+ for g, gb := range groupBys {
+ for a, agg := range aggs {
+ if gb == nil && agg.Func != modelv1.AggregationFunction_AGGREGATION_FUNCTION_SUM {
+ continue
+ }
+ for r := range ranges {
+ combos = append(combos, []int{f, g, a, r})
+ }
+ }
+ }
+ }
+ if raceEnabled {
+ combos = coverPairs(combos, nil, nil)
+ }
+ for _, c := range combos {
+ name, gb, agg, tr := fxNames[c[0]], groupBys[c[1]], aggs[c[2]], ranges[c[3]]
+ fx := fixtures[name]
+ q := &fxQuery{
+ tags: []string{fxTagSvc}, fields: []string{fxFldInt, fxFldFlt}, minTS: tr[0], maxTS: tr[1],
+ groupBy: gb, agg: agg, orderByTS: true, ascTS: true,
+ }
+ what := fmt.Sprintf("%s agg=%v groupBy=%v bucket=%v range=%v", name, agg.Func, gb != nil, gb != nil && gb.TimeBucket != nil, tr)
+ t.Run(what, func(t *testing.T) {
+ want := fx.run(t, q, true)
+ got := fx.run(t, q, false)
+ require.NotEmpty(t, want.points)
+ requireSameResult(t, want, got, what)
+ require.Positive(t, got.scan.mergeRows)
+ })
+ }
+}
+
+func TestTimeMergeMatchesAnIndependentModelForInterleavedSeries(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := oapFixture(t, 40)
+ order := fx.sids
+ for _, desc := range []bool{false, true} {
+ q := &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 20, orderByTS: true, ascTS: !desc}
+ res := fx.run(t, q, false)
+ // The model: every (series, ts) once at its newest version, ordered by time
+ // (reversed for desc) and, inside one timestamp, by series id.
+ type key struct{ sid, ts int64 }
+ newest := map[key]int64{}
+ for _, part := range fx.written {
+ for _, r := range part {
+ for ts := r.from; ts <= r.to; ts++ {
+ k := key{int64(r.sid), ts}
+ newest[k] = max(newest[k], r.version)
+ }
+ }
+ }
+ var want []key
+ for k := range newest {
+ want = append(want, k)
+ }
+ slices.SortFunc(want, func(a, b key) int {
+ if a.ts != b.ts {
+ if desc {
+ return int(b.ts - a.ts)
+ }
+ return int(a.ts - b.ts)
+ }
+ return int(a.sid - b.sid)
+ })
+ require.Len(t, res.points, len(want), "desc=%v", desc)
+ for i, dp := range res.points {
+ d := dp.GetDataPoint()
+ require.Equal(t, want[i], key{int64(d.GetSid()), d.GetTimestamp().AsTime().UnixNano()}, "row %d desc=%v", i, desc)
+ require.Equal(t, newest[want[i]], d.GetVersion(), "row %d keeps the newest version", i)
+ }
+ require.Len(t, order, 40)
+ }
+}
+
+// TestTimeMergeDedupsDuplicatesAcrossPageBoundaries pins the one place the merge
+// deliberately differs from the legacy batch merge. mergeBatch stopped a page at
+// 4,096 rows wherever it stood, so a timestamp duplicated across parts that fell
+// on the cut was emitted twice, once per page; the row merge (queryResult.merge,
+// no page cap) and the independent model emit it once.
+func TestTimeMergeDedupsDuplicatesAcrossPageBoundaries(t *testing.T) {
+ verifyNoScanLeaks(t)
+ // Row i of the series is ts i. The legacy page ends after row 4,096: the newer
+ // part's ts 4096 wins the page, the older part's ts 4096 starts the next one.
+ fx := newFixture(t, fxPart{{1, 1, 9000, 1}}, fxPart{{1, 4096, 6000, 2}})
+ q := &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 20, ascTS: true, order: []common.SeriesID{1}}
+
+ // The row path is the reference for what a query returns.
+ schema := q.schema(t)
+ qr := fx.newResult(t, q, schema)
+ var rowPathTS []int64
+ for {
+ r := qr.Pull()
+ if r == nil {
+ break
+ }
+ require.NoError(t, r.Error)
+ rowPathTS = append(rowPathTS, r.Timestamps...)
+ }
+ qr.Release()
+ require.Len(t, rowPathTS, 9000)
+ require.True(t, slices.IsSorted(rowPathTS))
+ require.Len(t, slices.Compact(slices.Clone(rowPathTS)), 9000, "the row path never repeats a timestamp")
+
+ legacy := fx.run(t, q, true)
+ got := fx.run(t, q, false)
+ require.Len(t, got.points, 9000)
+ for i, dp := range got.points {
+ require.EqualValues(t, rowPathTS[i], dp.GetDataPoint().GetTimestamp().AsTime().UnixNano())
+ }
+ require.Len(t, legacy.points, 9001, "the legacy batch merge emits the duplicate that straddles its 4,096-row page")
+ var dups []int64
+ for i := 1; i < len(legacy.points); i++ {
+ a, b := legacy.points[i-1].GetDataPoint(), legacy.points[i].GetDataPoint()
+ if a.GetTimestamp().AsTime().UnixNano() == b.GetTimestamp().AsTime().UnixNano() {
+ dups = append(dups, a.GetTimestamp().AsTime().UnixNano())
+ require.Greater(t, a.GetVersion(), b.GetVersion(), "the newest version came first")
+ }
+ }
+ require.Equal(t, []int64{4096}, dups)
+ // Everywhere else the two agree: drop the stray row and the results match.
+ trimmed := fxResult{points: append(slices.Clone(legacy.points[:4096]), legacy.points[4097:]...)}
+ for i := range got.points {
+ require.True(t, proto.Equal(trimmed.points[i], got.points[i]), "row %d", i)
+ }
+}
+
+func TestTimeMergeSetsSeriesBoundariesForInterleavedRuns(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := oapFixture(t, 6)
+ q := &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 20, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ qr := fx.newResult(t, q, schema)
+ defer qr.Release()
+ rows := 0
+ for {
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ break
+ }
+ rows += b.RowCount()
+ require.NotEmpty(t, b.SeriesBoundaries, "rows of several series interleave in this batch")
+ require.Equal(t, b.RowCount(), b.SeriesBoundaries[len(b.SeriesBoundaries)-1])
+ start := 0
+ for _, end := range b.SeriesBoundaries {
+ require.Greater(t, end, start)
+ for r := start; r < end; r++ {
+ require.Equal(t, b.SeriesIDs[start], b.SeriesIDs[r], "a run is one series")
+ }
+ if end < b.RowCount() {
+ require.NotEqual(t, b.SeriesIDs[end-1], b.SeriesIDs[end], "runs are maximal")
+ }
+ start = end
+ }
+ b.Release()
+ }
+ require.Positive(t, rows)
+}
+
+// TestTimeMergeDecodesCursorsAtTheFrontier is the memory bound: series that all
+// cover the same time range, each flushed in many parts that are disjoint in time,
+// leave one block per series live at any instant, not one per part.
+func TestTimeMergeDecodesCursorsAtTheFrontier(t *testing.T) {
+ verifyNoScanLeaks(t)
+ const (
+ series = frontierSeries
+ blocks = 12
+ span = 200
+ )
+ parts := make([]fxPart, blocks)
+ for p := range parts {
+ for sid := common.SeriesID(1); sid <= series; sid++ {
+ parts[p] = append(parts[p], fxRange{sid, int64(p*span + 1), int64((p + 1) * span), int64(p + 1)})
+ }
+ }
+ fx := newFixture(t, parts...)
+ q := &fxQuery{tags: []string{fxTagURL, fxTagSvc}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 20, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ for _, desc := range []bool{false, true} {
+ q.ascTS = !desc
+ qr := fx.newResult(t, q, schema)
+ qr.dp = nil // the test reads cursor state between pulls, which decodes ahead would race with
+ all := slices.Clone(qr.data)
+ require.Len(t, all, series*blocks)
+ decodedAndHeld := func() (held, returned int) {
+ for _, bc := range all {
+ switch {
+ case bc.refs.Load() > 0 && len(bc.timestamps) > 0:
+ held++
+ case bc.refs.Load() == 0:
+ returned++
+ }
+ }
+ return held, returned
+ }
+ maxHeld, rows, batches := 0, 0, 0
+ returnedMidScan := 0
+ for {
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ break
+ }
+ rows += b.RowCount()
+ batches++
+ b.Release()
+ held, returned := decodedAndHeld()
+ maxHeld = max(maxHeld, held)
+ if rows < series*blocks*span/2 {
+ returnedMidScan = max(returnedMidScan, returned)
+ }
+ }
+ require.Equal(t, series*blocks*span, rows)
+ require.Greater(t, batches, 4)
+ stats := qr.scan.stats
+ require.EqualValues(t, series*blocks, stats.mergeCursors, "every block is decoded once, desc=%v", desc)
+ require.LessOrEqual(t, int(stats.mergePeakLive), series+series/4+minFinishedLimit, "one live block per series at the frontier, desc=%v", desc)
+ t.Logf("peak live %d held %d (series %d, blocks per series %d)", stats.mergePeakLive, maxHeld, series, blocks)
+ require.GreaterOrEqual(t, int(stats.mergePeakLive), series)
+ require.LessOrEqual(t, maxHeld, series+series/4+minFinishedLimit, "decoded blocks held between pulls, desc=%v", desc)
+ require.Greater(t, returnedMidScan, series*(blocks/2-1), "cursors went back to the pool while the scan was still running, desc=%v", desc)
+ qr.Release()
+ for _, bc := range all {
+ require.EqualValues(t, 0, bc.refs.Load(), "every cursor is back in the pool at the end")
+ }
+ }
+}
+
+// TestTimeMergeLoaderIsThePlugInPoint checks the hook step 4b needs: every decode
+// goes through windowScan.load, in the order the frontier reaches the cursors.
+func TestTimeMergeLoaderIsThePlugInPoint(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := newFixture(t, fxPart{{1, 1, 100, 1}, {2, 50, 150, 1}}, fxPart{{1, 90, 200, 2}, {2, 140, 300, 2}})
+ q := &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1000, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ qr := fx.newResult(t, q, schema)
+ defer qr.Release()
+ qr.dp = nil // the order decodes happen in is what is asserted
+ require.True(t, qr.useWindowScan())
+ var loaded [][2]int64
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ loaded = append(loaded, [2]int64{int64(bc.bm.seriesID), bc.bm.timestamps.min})
+ return decodeCursor(ctx, bc, desc)
+ }
+ before := len(loaded)
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ b.Release()
+ require.Greater(t, len(loaded), before)
+ for {
+ b, err = qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ break
+ }
+ b.Release()
+ }
+ require.Equal(t, [][2]int64{{1, 1}, {2, 50}, {1, 90}, {2, 140}}, loaded, "cursors are decoded in frontier order")
+}
+
+func TestTimeMergeReleasesEverythingOnCancelAndEarlyClose(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := oapFixture(t, 30)
+ q := &fxQuery{tags: allTags, fields: allFields, minTS: 0, maxTS: 1 << 20, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ before := blockCursorPool.RefsCount()
+
+ qr := fx.newResult(t, q, schema)
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ b.Release()
+ qr.Release()
+ require.Equal(t, before, blockCursorPool.RefsCount(), "an early release returns every cursor")
+
+ ctx, cancel := context.WithCancel(context.Background())
+ qr = fx.newResult(t, q, schema)
+ qr.ctx = ctx
+ b, err = qr.PullBatch(ctx)
+ require.NoError(t, err)
+ b.Release()
+ cancel()
+ _, err = qr.PullBatch(ctx)
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "interrupt")
+ qr.Release()
+ require.Equal(t, before, blockCursorPool.RefsCount())
+
+ // A cancel in the middle of a merge page: the page's batch is discarded.
+ ctx, cancel = context.WithCancel(context.Background())
+ qr = fx.newResult(t, q, schema)
+ qr.ctx = ctx
+ qr.dp = nil
+ require.True(t, qr.useWindowScan())
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ cancel()
+ return decodeCursor(ctx, bc, desc)
+ }
+ _, err = qr.PullBatch(ctx)
+ require.Error(t, err)
+ qr.Release()
+ require.Equal(t, before, blockCursorPool.RefsCount())
+}
+
+// TestTimeMergeBatchesBorrowFromCursorsUnderPoisoning reads the merged string,
+// bytes and array cells after the query is released and the cursors that were
+// consumed are back in the pool: the batch's pins must keep exactly the memory it
+// borrows alive.
+func TestTimeMergeBatchesBorrowFromCursorsUnderPoisoning(t *testing.T) {
+ verifyNoScanLeaks(t)
+ defer setPoisonReleasedCursor(setPoisonReleasedCursor(true))
+ fx := oapFixture(t, 12)
+ q := &fxQuery{tags: []string{fxTagURL, fxTagArr, fxTagBin, fxTagSvc}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 20, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ qr := fx.newResult(t, q, schema)
+ var held []*model.MeasureBatch
+ for {
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ break
+ }
+ held = append(held, b)
+ }
+ qr.Release()
+ for _, b := range held {
+ require.NotZero(t, b.Owners.Len(), "string cells borrow, so the batch pins its cursors")
+ for r := range b.RowCount() {
+ url := b.Tags[0].(interface{ Data() []string }).Data()[r]
+ want := fmt.Sprintf("/u/%d/%d/%d", b.SeriesIDs[r], b.Timestamps[r], b.Versions[r])
+ require.Equal(t, want, url)
+ }
+ b.Release()
+ }
+}
+
+// TestTimeMergeKeepsTheNewestVersionOfDuplicatesInsideOneBlock covers the
+// replace branch of the duplicate rule: rows of one timestamp that sit in one
+// block in ascending version order reach the merge oldest first.
+func TestTimeMergeKeepsTheNewestVersionOfDuplicatesInsideOneBlock(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := newFixture(t,
+ fxPart{{1, 1, 40, 1}, {1, 20, 60, 2}, {1, 30, 50, 3}, {2, 1, 40, 1}, {2, 20, 60, 2}},
+ fxPart{{2, 35, 70, 4}},
+ )
+ for _, asc := range []bool{true, false} {
+ q := &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1000, orderByTS: true, ascTS: asc}
+ want := fx.run(t, q, true)
+ got := fx.run(t, q, false)
+ requireSameResult(t, want, got, fmt.Sprintf("asc=%v", asc))
+ newest := map[[2]int64]int64{}
+ for _, part := range fx.written {
+ for _, r := range part {
+ for ts := r.from; ts <= r.to; ts++ {
+ k := [2]int64{int64(r.sid), ts}
+ newest[k] = max(newest[k], r.version)
+ }
+ }
+ }
+ require.Len(t, got.points, len(newest))
+ for _, dp := range got.points {
+ d := dp.GetDataPoint()
+ require.Equal(t, newest[[2]int64{int64(d.GetSid()), d.GetTimestamp().AsTime().UnixNano()}], d.GetVersion())
+ }
+ }
+}
+
+// syntheticCursor is a decoded cursor with only keys: every projected column is
+// missing, so the output is null cells, which the legacy merge fills the same way.
+func syntheticCursor(sid common.SeriesID, ts, versions []int64) *blockCursor {
+ bc := generateBlockCursor()
+ bc.bm.seriesID = sid
+ bc.bm.timestamps.min, bc.bm.timestamps.max = ts[0], ts[len(ts)-1]
+ bc.minTimestamp, bc.maxTimestamp = 0, 1<<40
+ bc.timestamps = append(bc.timestamps, ts...)
+ bc.versions = append(bc.versions, versions...)
+ return bc
+}
+
+// TestTimeMergeKeysMatchTheLegacyHeapMergeOnUnsortedVersions feeds the merge the
+// streams only a hand-built cursor can: equal timestamps inside one cursor with
+// versions in ascending order, so the legacy replace rule (a higher version
+// overwrites the row just emitted) decides the winner.
+func TestTimeMergeKeysMatchTheLegacyHeapMergeOnUnsortedVersions(t *testing.T) {
+ verifyNoScanLeaks(t)
+ q := &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: 0, maxTS: 1 << 40, orderByTS: true, ascTS: true}
+ schema := q.schema(t)
+ type spec struct {
+ ts, versions []int64
+ sid common.SeriesID
+ }
+ specs := []spec{
+ {sid: 1, ts: []int64{5, 5, 5, 7, 9}, versions: []int64{1, 3, 2, 1, 1}},
+ {sid: 1, ts: []int64{5, 6, 7, 7}, versions: []int64{2, 1, 4, 2}},
+ {sid: 2, ts: []int64{5, 5, 8}, versions: []int64{9, 10, 1}},
+ }
+ build := func() []*blockCursor {
+ var out []*blockCursor
+ for _, sp := range specs {
+ out = append(out, syntheticCursor(sp.sid, sp.ts, sp.versions))
+ }
+ return out
+ }
+ for _, desc := range []bool{false, true} {
+ cursors := build()
+ if desc {
+ for i, sp := range specs {
+ slices.Reverse(cursors[i].timestamps)
+ slices.Reverse(cursors[i].versions)
+ cursors[i].idx = len(sp.ts) - 1
+ }
+ }
+ qr := &queryResult{ctx: context.Background(), batchSchema: schema, orderByTS: true, ascTS: !desc, sidToIndex: map[common.SeriesID]int{}}
+ qr.data = cursors
+ wantBatch, err := qr.mergeBatchFromHeap()
+ require.NoError(t, err)
+ var wantTS, wantVer []int64
+ var wantSID []common.SeriesID
+ for wantBatch != nil {
+ wantTS = append(wantTS, wantBatch.Timestamps...)
+ wantVer = append(wantVer, wantBatch.Versions...)
+ wantSID = append(wantSID, wantBatch.SeriesIDs...)
+ wantBatch.Release()
+ wantBatch, err = qr.mergeBatchFromHeap()
+ require.NoError(t, err)
+ }
+
+ cursors = build()
+ if desc {
+ for i, sp := range specs {
+ slices.Reverse(cursors[i].timestamps)
+ slices.Reverse(cursors[i].versions)
+ cursors[i].idx = len(sp.ts) - 1
+ }
+ }
+ qr = &queryResult{ctx: context.Background(), batchSchema: schema, orderByTS: true, ascTS: !desc}
+ var stats scanStats
+ load := func(_ context.Context, _ *blockCursor, _ bool) (bool, error) { return true, nil }
+ m := newTimeMerge(qr, &stats, load, newColPlan(schema), cursors, desc, nil)
+ var gotTS, gotVer []int64
+ var gotSID []common.SeriesID
+ for {
+ b, mergeErr := m.nextPage(context.Background())
+ require.NoError(t, mergeErr)
+ if b == nil {
+ break
+ }
+ gotTS = append(gotTS, b.Timestamps...)
+ gotVer = append(gotVer, b.Versions...)
+ gotSID = append(gotSID, b.SeriesIDs...)
+ b.Release()
+ }
+ m.close()
+ require.NotEmpty(t, wantTS)
+ require.Equal(t, wantTS, gotTS, "desc=%v", desc)
+ require.Equal(t, wantVer, gotVer, "desc=%v", desc)
+ require.Equal(t, wantSID, gotSID, "desc=%v", desc)
+ }
+}
+
+// mergeBatchFromHeap runs one legacy mergeBatch page over cursors that are already
+// decoded, initializing the heap on first use.
+func (qr *queryResult) mergeBatchFromHeap() (*model.MeasureBatch, error) {
+ if !qr.loaded {
+ qr.loaded = true
+ heap.Init(qr)
+ }
+ return qr.mergeBatch(qr.storedIndexValue, qr.batchSchema)
+}
diff --git a/banyand/measure/scan_metrics_test.go b/banyand/measure/scan_metrics_test.go
new file mode 100644
index 000000000..b07bf5ba8
--- /dev/null
+++ b/banyand/measure/scan_metrics_test.go
@@ -0,0 +1,122 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "testing"
+
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/banyand/observability"
+ obsservice "github.com/apache/skywalking-banyandb/banyand/observability/services"
+ "github.com/apache/skywalking-banyandb/pkg/meter"
+ "github.com/apache/skywalking-banyandb/pkg/meter/prom"
+)
+
+// promRegistry is the part of observability.MetricsRegistry the supplier uses, backed
+// by a real Prometheus registry so the exposition can be read back.
+type promRegistry struct {
+ observability.MetricsRegistry
+ reg *prometheus.Registry
+}
+
+func (r *promRegistry) With(scope meter.Scope) observability.Factory {
+ return obsservice.NewFactory(prom.NewProvider(scope, r.reg), nil, nil)
+}
+
+// scanMetricNames is every metric the scan path v2 added to the measure query metrics.
+var scanMetricNames = []string{
+ "total_scan_windows", "total_scan_windows_overlapped", "total_scan_segments_fast", "total_scan_segments_overlap",
+ "total_scan_merge_rows", "total_scan_merge_cursors", "scan_merge_peak_live_cursors",
+ "scan_prefetch_depth", "scan_decode_queue_wait_seconds", "total_scan_reservations_waiting", "total_scan_reservations_failed",
+}
+
+// observeAllScanMetrics touches every scan metric once.
+func observeAllScanMetrics(qm *queryMetrics) {
+ qm.scanWindows.Inc(1)
+ qm.scanWindowsOverlapped.Inc(1)
+ qm.scanSegmentsFast.Inc(1)
+ qm.scanSegmentsOverlap.Inc(1)
+ qm.scanMergeRows.Inc(1)
+ qm.scanMergeCursors.Inc(1)
+ qm.scanMergePeakLive.Observe(1)
+ qm.scanPrefetchDepth.Observe(1)
+ qm.scanDecodeQueueWait.Observe(0.001)
+ qm.scanReservationsWaiting.Inc(1)
+ qm.scanReservationsFailed.Inc(1)
+}
+
+func exposedNames(t *testing.T, reg *prometheus.Registry) map[string]int {
+ t.Helper()
+ families, err := reg.Gather()
+ require.NoError(t, err)
+ out := map[string]int{}
+ for _, mf := range families {
+ out[mf.GetName()] = len(mf.GetMetric())
+ }
+ return out
+}
+
+func TestScanMetricsAreRegisteredOnceAndExposedWithTheQueryMetrics(t *testing.T) {
+ reg := prometheus.NewRegistry()
+ s := &supplier{omr: &promRegistry{reg: reg}}
+ // newMetrics panics on a duplicate registration, so a metric registered twice by
+ // the same call fails here.
+ s.newMetrics(common.Position{Module: "measure", Database: "g1"})
+ qm := s.queryMetricsOf("g1")
+ require.NotNil(t, qm)
+ observeAllScanMetrics(qm)
+ qm.totalQueryResultStarted.Inc(1)
+
+ got := exposedNames(t, reg)
+ for _, name := range scanMetricNames {
+ require.Contains(t, got, "banyandb_measure_query_"+name, "%s is exposed with the other measure query metrics", name)
+ }
+ // They share the scope of the metrics scan v2 builds on.
+ require.Contains(t, got, "banyandb_measure_query_total_query_result_started")
+}
+
+// TestScanMetricsOfAnEarlierGroupSurviveALaterGroupOpening guards the query metrics of
+// a node with several groups: opening a second group must not unregister the first
+// group's collectors.
+func TestScanMetricsOfAnEarlierGroupSurviveALaterGroupOpening(t *testing.T) {
+ reg := prometheus.NewRegistry()
+ s := &supplier{omr: &promRegistry{reg: reg}}
+ s.newMetrics(common.Position{Module: "measure", Database: "g1"})
+ first := s.queryMetricsOf("g1")
+ observeAllScanMetrics(first)
+ s.newMetrics(common.Position{Module: "measure", Database: "g2"})
+ observeAllScanMetrics(first) // a measure opened in g1 keeps its own pointer
+ observeAllScanMetrics(s.queryMetricsOf("g2"))
+ got := exposedNames(t, reg)
+ require.Equal(t, 2, got["banyandb_measure_query_total_scan_windows"], "one series per group")
+}
+
+func TestReopeningAGroupReplacesOnlyItsOwnQueryMetrics(t *testing.T) {
+ reg := prometheus.NewRegistry()
+ s := &supplier{omr: &promRegistry{reg: reg}}
+ s.newMetrics(common.Position{Module: "measure", Database: "g1"})
+ s.newMetrics(common.Position{Module: "measure", Database: "g2"})
+ s.newMetrics(common.Position{Module: "measure", Database: "g1"})
+ observeAllScanMetrics(s.queryMetricsOf("g1"))
+ observeAllScanMetrics(s.queryMetricsOf("g2"))
+ require.Equal(t, 2, exposedNames(t, reg)["banyandb_measure_query_total_scan_windows"])
+ require.Nil(t, s.queryMetricsOf("g3"))
+}
diff --git a/banyand/measure/scan_prefetch.go b/banyand/measure/scan_prefetch.go
new file mode 100644
index 000000000..a11555206
--- /dev/null
+++ b/banyand/measure/scan_prefetch.go
@@ -0,0 +1,334 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "sort"
+)
+
+// Prefetch (measure scan path v2, step 4b).
+//
+// The scan knows, before it reads a row, every cursor it will decode and the order
+// it will ask for them: units in order, segments of a window in order, and inside a
+// merge the cursors in the order of their first readable timestamp. The prefetcher
+// holds that sequence and asks the node's decodePool to decode the next few cursors
+// while the scan emits the current one. A decode is requested in sequence order and
+// collected in sequence order (take), so which worker finishes first never changes
+// what the scan sees.
+//
+// Depth is derived, not configured. It starts at prefetchMinDepth cursors ahead and
+// grows by one each time the scan has the room it asked for and every reservation
+// succeeded, up to a cap; a refused reservation stops the growth and pins the depth
+// at what the budget actually allowed, and prefetch waits: nothing further is
+// requested until a take (or the release of a reservation, seen at the next take)
+// makes room. A slow consumer therefore stalls decoding instead of growing memory.
+//
+// The cap is prefetchMaxAhead cursors, one per window of a scan that lends one block
+// per window. Inside a merge the cursors that become needed together scale with the
+// series in the frontier, so the cap follows the widest frontier seen: a quarter of
+// it, the same share finishedLimit lets a page hold on top of the frontier, so
+// prefetch adds at most about that much decoded memory to the merge's own peak. It
+// is limited to prefetchMaxAheadMerge; beyond that more decodes would only sit in
+// memory. The reservations, not the cap, are what ultimately bound the memory.
+const (
+ prefetchMinDepth = 2
+ prefetchMaxAhead = 8
+ prefetchMaxAheadMerge = 32
+)
+
+// prefetchItem is one cursor of the scan sequence.
+type prefetchItem struct {
+ bc *blockCursor
+ task decodeTask
+ res scanReservation
+ size uint64
+ // requested is true from the reservation until the scan takes the result.
+ requested bool
+ // promoted is true once the task was also queued as urgent.
+ promoted bool
+}
+
+// forget drops what the item still references once its decode ended, or was
+// withdrawn, so the scan's items do not keep every consumed cursor and its decoded
+// buffers reachable. The task's state stays: a stale queue entry may still look at it.
+func (it *prefetchItem) forget() {
+ it.bc = nil
+ it.requested = false
+ it.task.clear()
+}
+
+// prefetcher feeds one scan from the node's decode pool. It is used by one
+// goroutine, the scan's.
+type prefetcher struct {
+ ctx context.Context
+ pool *decodePool
+ budget *scanBudget
+ qm *queryMetrics
+ stats *scanStats
+ // load is the scan's decode function, read when a decode is requested.
+ load *cursorLoader
+ // wake is signaled by the scan's decodes as they end.
+ wake chan struct{}
+ items []prefetchItem
+ // head is the next item the scan takes; items [head, next) are requested.
+ head, next int
+ depth int
+ frontier int
+ peak int
+ desc bool
+ closed bool
+ // stalled is true from a refused reservation until a pump fits everything it
+ // asked for: the budget is full, so the scan should hand back what it only
+ // holds to coalesce (windowScan.next).
+ stalled bool
+}
+
+// newPrefetcher plans the decodes of cursors, in the order the scan will take them.
+// Without a pool, or with fewer than two cursors, nothing is requested ahead and
+// take decodes inline, still through the reservation.
+func newPrefetcher(ctx context.Context, pool *decodePool, budget *scanBudget, qm *queryMetrics, stats *scanStats,
+ load *cursorLoader, desc bool, cursors []*blockCursor,
+) *prefetcher {
+ if budget == nil {
+ budget = newScanBudget(-1, nil)
+ }
+ if len(cursors) < 2 {
+ pool = nil
+ }
+ pf := &prefetcher{
+ ctx: ctx, pool: pool, budget: budget, qm: qm, stats: stats, load: load, desc: desc,
+ depth: prefetchMinDepth,
+ items: make([]prefetchItem, len(cursors)),
+ }
+ if pool != nil {
+ pf.wake = make(chan struct{}, 1)
+ }
+ for i, bc := range cursors {
+ pf.items[i].bc = bc
+ pf.items[i].size = bc.bm.uncompressedSizeBytes
+ }
+ return pf
+}
+
+// orderPending sorts cursors the way a timeMerge orders its pending cursors: by the
+// first timestamp the merge can read, latest first for a descending scan, arrival
+// order breaking ties.
+func orderPending(cursors []*blockCursor, desc bool) []*blockCursor {
+ sorted := make([]*blockCursor, len(cursors))
+ copy(sorted, cursors)
+ sort.SliceStable(sorted, func(i, j int) bool {
+ a, b := mergeStart(sorted[i], desc), mergeStart(sorted[j], desc)
+ if desc {
+ return a > b
+ }
+ return a < b
+ })
+ return sorted
+}
+
+// maxAhead is the cap on the depth.
+func (pf *prefetcher) maxAhead() int {
+ return min(max(pf.frontier/4, prefetchMaxAhead), prefetchMaxAheadMerge)
+}
+
+// waiting reports whether prefetch is held back by the memory budget.
+func (pf *prefetcher) waiting() bool { return pf != nil && pf.stalled }
+
+// setFrontier tells the prefetcher how many cursors the running merge holds in
+// its heap; 0 says no merge is running.
+func (pf *prefetcher) setFrontier(n int) {
+ if pf != nil {
+ pf.frontier = n
+ }
+}
+
+// pump requests decodes while there is room within the depth and the reservations
+// succeed.
+func (pf *prefetcher) pump() {
+ if pf.pool == nil || pf.closed {
+ return
+ }
+ for pf.next < len(pf.items) && pf.next-pf.head < min(pf.depth, pf.maxAhead()) {
+ it := &pf.items[pf.next]
+ if out := pf.budget.tryReserve(it.size, true, &it.res); out != reserveOK {
+ pf.stalled = true
+ pf.stats.reservationWaits++
+ if pf.qm != nil {
+ pf.qm.scanReservationsWaiting.Inc(1)
+ }
+ // What the budget allows is the depth; the next successful pump grows it.
+ pf.depth = max(prefetchMinDepth, pf.next-pf.head)
+ pf.peak = max(pf.peak, pf.next-pf.head)
+ return
+ }
+ pf.submit(it, false)
+ pf.next++
+ }
+ pf.stalled = false
+ pf.peak = max(pf.peak, pf.next-pf.head)
+ if pf.next < len(pf.items) && pf.depth < pf.maxAhead() {
+ pf.depth++
+ }
+}
+
+// submit hands the reserved item to the pool; urgent is for the item the scan is
+// waiting for.
+func (pf *prefetcher) submit(it *prefetchItem, urgent bool) {
+ if it.size > 0 {
+ it.bc.res = &it.res
+ }
+ it.requested, it.promoted = true, urgent
+ it.task.init(pf.ctx, *pf.load, it.bc, pf.desc, pf.qm, pf.wake)
+ pf.pool.submit(&it.task, urgent)
+}
+
+// take returns the decode of bc, which must be the next cursor of the sequence. It
+// has the contract of a cursorLoader: the caller keeps bc and releases it on false
+// or an error. A cursor out of sequence is decoded inline.
+func (pf *prefetcher) take(ctx context.Context, bc *blockCursor) (bool, error) {
+ if pf.closed || pf.head >= len(pf.items) || pf.items[pf.head].bc != bc {
+ return pf.loadDirect(ctx, bc)
+ }
+ it := &pf.items[pf.head]
+ if pf.pool == nil {
+ if err := pf.reserveCurrent(ctx, it); err != nil {
+ pf.advance()
+ return false, err
+ }
+ ok, err := (*pf.load)(ctx, bc, pf.desc)
+ pf.advance()
+ return ok, err
+ }
+ if !it.requested {
+ if err := pf.reserveCurrent(ctx, it); err != nil {
+ pf.advance()
+ return false, err
+ }
+ pf.submit(it, true)
+ pf.next = pf.head + 1
+ }
+ pf.promote(it)
+ pf.wait(ctx, &it.task)
+ ok, err := it.task.ok, it.task.err
+ pf.advance()
+ pf.pump()
+ return ok, err
+}
+
+// promote queues a requested task that has not started a second time, as urgent,
+// because the scan is now waiting for it: urgent tasks of every query are served
+// before any read-ahead, so without it the cursor a query needs next could starve
+// behind other queries' urgent decodes while it sits in the normal queue. The
+// stale entry left in the normal queue is harmless: decodeTask.run starts a task
+// only by moving it from queued to running, so whichever entry a worker pops
+// second finds the task started (or canceled) and does nothing.
+func (pf *prefetcher) promote(it *prefetchItem) {
+ if it.promoted || it.task.state.Load() != taskQueued {
+ return
+ }
+ it.promoted = true
+ pf.pool.submit(&it.task, true)
+}
+
+// discard gives up the head cursor without decoding it, when the scan stops before
+// asking for it. A decode already requested is withdrawn if it has not started, and
+// waited for if it has, so the caller can release bc safely.
+func (pf *prefetcher) discard(bc *blockCursor) {
+ if pf == nil || pf.closed || pf.head >= len(pf.items) || pf.items[pf.head].bc != bc {
+ return
+ }
+ it := &pf.items[pf.head]
+ if it.requested && !it.task.cancel(errScanClosed) {
+ pf.wait(pf.ctx, &it.task)
+ }
+ pf.advance()
+}
+
+// wait blocks until t has ended. If ctx ends first, a task that has not started is
+// withdrawn; one that has is still waited for, since the caller releases the cursor
+// as soon as take returns.
+func (pf *prefetcher) wait(ctx context.Context, t *decodeTask) {
+ ctxDone := ctx.Done()
+ for !t.finished() {
+ select {
+ case <-pf.wake:
+ case <-ctxDone:
+ ctxDone = nil
+ if t.cancel(ctx.Err()) {
+ return
+ }
+ }
+ }
+}
+
+// reserveCurrent reserves the memory the head item needs to be decoded now.
+func (pf *prefetcher) reserveCurrent(ctx context.Context, it *prefetchItem) error {
+ if err := pf.budget.reserveCurrent(ctx, it.size, &it.res); err != nil {
+ pf.stats.reservationFails++
+ if pf.qm != nil {
+ pf.qm.scanReservationsFailed.Inc(1)
+ }
+ return err
+ }
+ if it.size > 0 {
+ it.bc.res = &it.res
+ }
+ return nil
+}
+
+// advance moves past the head item; the scan owns its cursor from here.
+func (pf *prefetcher) advance() {
+ pf.items[pf.head].forget()
+ pf.head++
+ pf.next = max(pf.next, pf.head)
+}
+
+// loadDirect decodes a cursor that is not the next of the sequence, which the scan
+// never asks for; it keeps the accounting right if that ever changes.
+func (pf *prefetcher) loadDirect(ctx context.Context, bc *blockCursor) (bool, error) {
+ res := new(scanReservation)
+ if err := pf.budget.reserveCurrent(ctx, bc.bm.uncompressedSizeBytes, res); err != nil {
+ return false, err
+ }
+ bc.res = res
+ return (*pf.load)(ctx, bc, pf.desc)
+}
+
+// close withdraws every decode not yet taken and waits for those running, so that
+// no worker reads a cursor, or the parts behind it, once it returns. The cursors
+// stay with the scan, which releases them (and with them their reservations). It is
+// idempotent.
+func (pf *prefetcher) close() {
+ if pf == nil || pf.closed {
+ return
+ }
+ pf.closed = true
+ for i := pf.head; i < pf.next; i++ {
+ it := &pf.items[i]
+ if it.requested && !it.task.cancel(errScanClosed) {
+ pf.wait(pf.ctx, &it.task)
+ }
+ it.task.clear()
+ }
+ pf.items = nil
+ pf.stats.prefetchPeak = int64(pf.peak)
+ if pf.qm != nil && pf.peak > 0 {
+ pf.qm.scanPrefetchDepth.Observe(float64(pf.peak))
+ }
+}
diff --git a/banyand/measure/scan_prefetch_test.go b/banyand/measure/scan_prefetch_test.go
new file mode 100644
index 000000000..2808fc9c5
--- /dev/null
+++ b/banyand/measure/scan_prefetch_test.go
@@ -0,0 +1,1115 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "math/rand"
+ "runtime"
+ "slices"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/require"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/banyand/protector"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+// drainRows pulls the query dry and returns one line per row (series, timestamp,
+// version, the url tag and the int field), in emission order.
+func drainRows(t testing.TB, qr *queryResult) []string {
+ t.Helper()
+ rows, err := drainRowsErr(qr)
+ require.NoError(t, err)
+ return rows
+}
+
+// drainRowsErr is drainRows for goroutines other than the test's: it returns the
+// error of a failed pull instead of failing the test.
+func drainRowsErr(qr *queryResult) ([]string, error) {
+ var rows []string
+ for {
+ b, err := qr.PullBatch(context.Background())
+ if err != nil {
+ return nil, err
+ }
+ if b == nil {
+ return rows, nil
+ }
+ url := b.Tags[0].(*vectorized.TypedColumn[string]).Data()
+ fi := b.Fields[0].(*vectorized.TypedColumn[int64]).Data()
+ visit := func(r int) {
+ rows = append(rows, fmt.Sprintf("%d/%d/%d/%s/%d", b.SeriesIDs[r], b.Timestamps[r], b.Versions[r], url[r], fi[r]))
+ }
+ if b.Selection != nil {
+ for _, r := range b.Selection {
+ visit(int(r))
+ }
+ } else {
+ for r := range b.RowCount() {
+ visit(r)
+ }
+ }
+ b.Release()
+ }
+}
+
+// countGoroutines returns how many goroutines run a stack containing marker. The
+// suite cannot use a process-wide goroutine count: other tests leave goroutines of
+// their own coming and going, so a leak is looked for in the scan's own goroutines,
+// the pool's workers and the decodes they run.
+func countGoroutines(marker string) int {
+ buf := make([]byte, 1<<22)
+ buf = buf[:runtime.Stack(buf, true)]
+ n := 0
+ for _, g := range strings.Split(string(buf), "\n\n") {
+ if strings.Contains(g, marker) {
+ n++
+ }
+ }
+ return n
+}
+
+const (
+ workerMarker = "measure.(*decodePool).run("
+ taskMarker = "measure.(*decodeTask).run("
+)
+
+// sharedWorkers is the size of the fixture pool.
+const sharedWorkers = 4
+
+// onlyTheSharedPool starts the fixture pool and waits until it is the only pool
+// with workers, so a test can count its own pools' workers exactly.
+func onlyTheSharedPool(t testing.TB) {
+ t.Helper()
+ sharedFixturePool()
+ waitWorkers(t, sharedWorkers)
+}
+
+// waitWorkers waits until exactly want decode workers are running.
+func waitWorkers(t testing.TB, want int) {
+ t.Helper()
+ require.Eventually(t, func() bool { return countGoroutines(workerMarker) == want }, 5*time.Second, 5*time.Millisecond,
+ "want %d decode workers running", want)
+}
+
+// waitNoDecodes waits until no worker is inside a decode.
+func waitNoDecodes(t testing.TB) {
+ t.Helper()
+ require.Eventually(t, func() bool { return countGoroutines(taskMarker) == 0 }, 5*time.Second, 5*time.Millisecond,
+ "a decode is still running")
+}
+
+// manySeries is one part with n series of 1..rows rows each: n windows of one block.
+func manySeries(t testing.TB, n int, rows int64) *fxFixture {
+ var p fxPart
+ for sid := common.SeriesID(1); sid <= common.SeriesID(n); sid++ {
+ p = append(p, fxRange{sid, 1, rows, 1})
+ }
+ return newFixture(t, p)
+}
+
+func prefetchQuery(minTS, maxTS int64, orderByTS, asc bool) *fxQuery {
+ return &fxQuery{tags: []string{fxTagURL}, fields: []string{fxFldInt}, minTS: minTS, maxTS: maxTS, orderByTS: orderByTS, ascTS: asc}
+}
+
+// maxBlockBytes is the largest uncompressed block size among the query's cursors.
+func maxBlockBytes(qr *queryResult) int64 {
+ var size uint64
+ for _, bc := range qr.data {
+ size = max(size, bc.bm.uncompressedSizeBytes)
+ }
+ return int64(size)
+}
+
+// TestPrefetchMatchesSynchronousDecodeUnderRandomCompletionOrder runs queries of
+// every scan shape with the decode pool, a loader that finishes later-requested
+// decodes first, and checks the rows match the synchronous scan exactly.
+func TestPrefetchMatchesSynchronousDecodeUnderRandomCompletionOrder(t *testing.T) {
+ verifyNoScanLeaks(t)
+ mixed := mixedFixture(t)
+ oapSeries, stagSeries := 24, 12
+ if raceEnabled {
+ oapSeries, stagSeries = 12, 8
+ }
+ oap := oapFixture(t, oapSeries)
+ stag := staggeredFixture(t, stagSeries)
+ tests := []struct {
+ fx *fxFixture
+ q *fxQuery
+ }{
+ {mixed, prefetchQuery(0, 30000, false, true)},
+ {mixed, prefetchQuery(120, 700, true, true)},
+ {mixed, prefetchQuery(0, 30000, true, false)},
+ {oap, prefetchQuery(0, 1<<20, true, true)},
+ {oap, prefetchQuery(0, 1<<20, true, false)},
+ {oap, prefetchQuery(0, 1<<20, false, true)},
+ {stag, prefetchQuery(0, 1<<20, true, false)},
+ {stag, prefetchQuery(0, 1<<20, true, true)},
+ }
+ for i, tt := range tests {
+ t.Run(fmt.Sprintf("query %d", i), func(t *testing.T) {
+ schema := tt.q.schema(t)
+ ref := tt.fx.newResult(t, tt.q, schema)
+ ref.dp = nil
+ want := drainRows(t, ref)
+ ref.Release()
+ require.NotEmpty(t, want)
+
+ pool := newDecodePool(8, nil)
+ defer pool.close()
+ for seed := range 3 {
+ qr := tt.fx.newResult(t, tt.q, schema)
+ qr.dp = pool
+ require.True(t, qr.useWindowScan())
+ var started atomic.Int64
+ var mu sync.Mutex
+ var startOrder, finishOrder []int64
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ n := started.Add(1)
+ mu.Lock()
+ startOrder = append(startOrder, n)
+ mu.Unlock()
+ // Earlier decodes sleep longer, with a seeded jitter on top, so decodes finish
+ // in a different order than they were requested in.
+ jitter := rand.New(rand.NewSource(int64(seed)*1000 + n)).Intn(400) //nolint:gosec // deterministic jitter, not a security context
+ time.Sleep(time.Duration(8-(n%8))*300*time.Microsecond + time.Duration(jitter)*time.Microsecond)
+ ok, err := decodeCursor(ctx, bc, desc)
+ mu.Lock()
+ finishOrder = append(finishOrder, n)
+ mu.Unlock()
+ return ok, err
+ }
+ got := drainRows(t, qr)
+ require.Equal(t, want, got, "seed %d: rows differ from the synchronous scan", seed)
+ if len(startOrder) >= 8 {
+ require.False(t, slices.IsSorted(finishOrder), "seed %d: decodes should have completed out of request order", seed)
+ }
+ qr.Release()
+ }
+ })
+ }
+}
+
+// stubCursors returns n metadata-only cursors of the given uncompressed size.
+func stubCursors(n int, size uint64) []*blockCursor {
+ cursors := make([]*blockCursor, n)
+ for i := range cursors {
+ bc := generateBlockCursor()
+ bc.bm.seriesID = common.SeriesID(i + 1)
+ bc.bm.uncompressedSizeBytes = size
+ cursors[i] = bc
+ }
+ return cursors
+}
+
+func instantLoad(_ context.Context, _ *blockCursor, _ bool) (bool, error) { return true, nil }
+
+func TestPrefetchDepthGrowsWhileReservationsSucceedAndStopsAtTheCap(t *testing.T) {
+ verifyNoScanLeaks(t)
+ pool := newDecodePool(4, nil)
+ defer pool.close()
+ cursors := stubCursors(64, 100)
+ defer releaseCursors(cursors)
+ var stats scanStats
+ load := cursorLoader(instantLoad)
+ pf := newPrefetcher(context.Background(), pool, newScanBudget(-1, nil), nil, &stats, &load, false, cursors)
+
+ pf.pump()
+ require.Equal(t, 2, pf.next-pf.head, "a scan starts two cursors ahead")
+ last := pf.depth
+ maxDepth := last
+ for i, bc := range cursors {
+ ok, err := pf.take(context.Background(), bc)
+ require.NoError(t, err)
+ require.True(t, ok)
+ require.GreaterOrEqual(t, pf.depth, last, "depth never shrinks while reservations succeed (take %d)", i)
+ require.LessOrEqual(t, pf.next-pf.head, prefetchMaxAhead, "never more than the cap ahead")
+ last = pf.depth
+ maxDepth = max(maxDepth, last)
+ }
+ require.Equal(t, prefetchMaxAhead, maxDepth, "depth grows up to the cap and stops there")
+ pf.close()
+ require.EqualValues(t, prefetchMaxAhead, stats.prefetchPeak)
+ require.Zero(t, stats.reservationWaits)
+}
+
+func TestPrefetchCapFollowsTheMergeFrontierUpToItsLimit(t *testing.T) {
+ verifyNoScanLeaks(t)
+ pool := newDecodePool(2, nil)
+ defer pool.close()
+ cursors := stubCursors(200, 10)
+ defer releaseCursors(cursors)
+ var stats scanStats
+ load := cursorLoader(instantLoad)
+ pf := newPrefetcher(context.Background(), pool, newScanBudget(-1, nil), nil, &stats, &load, false, cursors)
+ require.Equal(t, prefetchMaxAhead, pf.maxAhead())
+ pf.setFrontier(80)
+ require.Equal(t, 20, pf.maxAhead())
+ pf.setFrontier(20)
+ require.Equal(t, prefetchMaxAhead, pf.maxAhead(), "a small frontier keeps the window cap")
+ pf.setFrontier(500)
+ require.Equal(t, prefetchMaxAheadMerge, pf.maxAhead())
+ pf.setFrontier(0)
+ require.Equal(t, prefetchMaxAhead, pf.maxAhead())
+ pf.pump()
+ pf.setFrontier(500)
+ for _, bc := range cursors[:100] {
+ _, err := pf.take(context.Background(), bc)
+ require.NoError(t, err)
+ }
+ require.LessOrEqual(t, pf.next-pf.head, prefetchMaxAheadMerge)
+ require.Greater(t, pf.next-pf.head, prefetchMaxAhead, "a wide merge frontier lets prefetch run past the window cap")
+ pf.close()
+}
+
+func TestPrefetchWaitsUnderATightBudgetAndStillCompletes(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 30, 100)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ pool := newDecodePool(4, nil)
+ defer pool.close()
+
+ ref := fx.newResult(t, q, schema)
+ ref.dp = nil
+ want := drainRows(t, ref)
+ ref.Release()
+
+ qr := fx.newResult(t, q, schema)
+ block := maxBlockBytes(qr)
+ budget := newScanBudget(block*2+block/2, pool)
+ qr.dp, qr.budget = pool, budget
+ got := drainRows(t, qr)
+ require.Equal(t, want, got, "a tight budget slows the scan down, it does not change it")
+ stats := qr.scan.stats
+ qr.Release()
+ require.Positive(t, stats.reservationWaits, "prefetch waited for room")
+ require.Zero(t, stats.reservationFails, "the cursor the scan needed was always reservable")
+ require.LessOrEqual(t, stats.prefetchPeak, int64(2), "two blocks fit, so at most the current and one ahead")
+ require.Zero(t, budget.held.Load(), "every reservation is back")
+ require.Zero(t, pool.reserved.Load())
+}
+
+func TestScanFailsWithTheQuotaErrorOnlyWhenTheCurrentCursorCannotBeReserved(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 6, 300)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ pool := newDecodePool(2, nil)
+ defer pool.close()
+ before := blockCursorPool.RefsCount()
+
+ // Less than one block: the very first cursor cannot be reserved.
+ qr := fx.newResult(t, q, schema)
+ block := maxBlockBytes(qr)
+ budget := newScanBudget(block/2, pool)
+ qr.dp, qr.budget = pool, budget
+ _, err := qr.PullBatch(context.Background())
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "block scan quota exceeded: used")
+ require.Contains(t, err.Error(), fmt.Sprintf("quota is %d bytes", block/2))
+ require.EqualValues(t, 1, qr.scan.stats.reservationFails)
+ qr.Release()
+ require.Zero(t, budget.held.Load())
+ require.Equal(t, before, blockCursorPool.RefsCount())
+
+ // Room for one and a half blocks: the first batch is read and held by the
+ // consumer, so its reservation stays and the second cursor cannot be reserved.
+ qr = fx.newResult(t, q, schema)
+ budget = newScanBudget(block+block/2, pool)
+ qr.dp, qr.budget = pool, budget
+ first, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ require.NotNil(t, first)
+ _, err = qr.PullBatch(context.Background())
+ require.Error(t, err)
+ require.Contains(t, err.Error(), "block scan quota exceeded")
+ require.EqualValues(t, 1, qr.scan.stats.reservationFails)
+ _, again := qr.PullBatch(context.Background())
+ require.Error(t, again, "the error is sticky")
+ qr.Release()
+ require.Positive(t, budget.held.Load(), "the batch the consumer holds still pins its cursor")
+ first.Release()
+ require.Zero(t, budget.held.Load())
+ require.Zero(t, pool.reserved.Load())
+ require.Equal(t, before, blockCursorPool.RefsCount())
+
+ // Releasing each batch is enough for the same budget to finish the scan.
+ qr = fx.newResult(t, q, schema)
+ budget = newScanBudget(block+block/2, pool)
+ qr.dp, qr.budget = pool, budget
+ require.Len(t, drainRows(t, qr), 6*300)
+ qr.Release()
+ require.Zero(t, budget.held.Load())
+}
+
+func TestScanReservationIsHeldUntilTheBorrowingBatchIsReleased(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 1, 300)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ pool := newDecodePool(2, nil)
+ defer pool.close()
+ before := blockCursorPool.RefsCount()
+
+ qr := fx.newResult(t, q, schema)
+ block := maxBlockBytes(qr)
+ require.Positive(t, block)
+ budget := newScanBudget(1<<30, pool)
+ qr.dp, qr.budget = pool, budget
+
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ require.NotNil(t, b)
+ require.EqualValues(t, block, budget.held.Load(), "a block pinned by several columns is counted once")
+ require.EqualValues(t, block, pool.reserved.Load())
+ // The scan drops its own reference as soon as it has lent the block.
+ next, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ require.Nil(t, next)
+ require.EqualValues(t, block, budget.held.Load(), "the batch still borrows the block")
+ qr.Release()
+ require.EqualValues(t, block, budget.held.Load(), "releasing the query does not free what a batch borrows")
+ b.Release()
+ require.Zero(t, budget.held.Load(), "the last borrower frees the reservation")
+ require.Zero(t, pool.reserved.Load())
+ require.Zero(t, budget.tracker.Used())
+ require.Equal(t, before, blockCursorPool.RefsCount())
+}
+
+func TestScanReleasesReservationsOfPrefetchedCursorsOnEarlyClose(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 40, 100)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ onlyTheSharedPool(t)
+ pool := newDecodePool(4, nil)
+ defer pool.close()
+ waitWorkers(t, sharedWorkers+4)
+ before := blockCursorPool.RefsCount()
+
+ qr := fx.newResult(t, q, schema)
+ budget := newScanBudget(1<<30, pool)
+ qr.dp, qr.budget = pool, budget
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ b.Release()
+ require.Positive(t, qr.scan.pf.next-qr.scan.pf.head, "decodes are in flight or done ahead of the scan")
+ require.Greater(t, budget.held.Load(), int64(0))
+ qr.Release() // what a LIMIT does after its last row
+ require.Zero(t, budget.held.Load())
+ require.Zero(t, pool.reserved.Load())
+ require.Equal(t, before, blockCursorPool.RefsCount(), "every cursor is back in the pool")
+ waitNoDecodes(t)
+}
+
+func TestScanCancellationWhileDecodesAreInFlight(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 40, 300)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ onlyTheSharedPool(t)
+ pool := newDecodePool(4, nil)
+ waitWorkers(t, sharedWorkers+4)
+ before := blockCursorPool.RefsCount()
+
+ ctx, cancel := context.WithCancel(context.Background())
+ qr := fx.newResult(t, q, schema)
+ budget := newScanBudget(1<<30, pool)
+ qr.dp, qr.budget, qr.ctx = pool, budget, ctx
+ require.True(t, qr.useWindowScan())
+ gate := make(chan struct{})
+ var running atomic.Int32
+ first := qr.scan.pf.items[0].bc
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ // The first cursor of the plan, which the scan waits for, decodes at once; every
+ // decode requested ahead of it blocks inside the loader, as a slow disk read
+ // would. The cursor is chosen by identity: workers may start in any order.
+ if bc != first {
+ running.Add(1)
+ <-gate
+ }
+ return decodeCursor(ctx, bc, desc)
+ }
+ b, err := qr.PullBatch(ctx)
+ require.NoError(t, err)
+ b.Release()
+ require.Eventually(t, func() bool { return running.Load() > 0 }, 5*time.Second, time.Millisecond)
+ require.Positive(t, budget.held.Load(), "the decodes in flight hold their reservations")
+ cancel()
+ released := make(chan struct{})
+ go func() {
+ qr.Release()
+ close(released)
+ }()
+ select {
+ case <-released:
+ t.Fatal("Release returned while a decode was still reading the cursor")
+ case <-time.After(100 * time.Millisecond):
+ }
+ close(gate)
+ select {
+ case <-released:
+ case <-time.After(5 * time.Second):
+ t.Fatal("Release did not return after the in-flight decode finished")
+ }
+ require.Zero(t, budget.held.Load())
+ require.Zero(t, pool.reserved.Load())
+ require.Equal(t, before, blockCursorPool.RefsCount())
+ pool.close()
+ waitWorkers(t, sharedWorkers)
+}
+
+func TestScanCancelBetweenPullsWaitsForTheRunningPrefetchedDecode(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 40, 300)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ onlyTheSharedPool(t)
+ pool := newDecodePool(4, nil)
+ waitWorkers(t, sharedWorkers+4)
+ before := blockCursorPool.RefsCount()
+
+ qr := fx.newResult(t, q, schema)
+ budget := newScanBudget(1<<30, pool)
+ qr.dp, qr.budget = pool, budget
+ require.True(t, qr.useWindowScan())
+ gate := make(chan struct{})
+ var running atomic.Int32
+ var released atomic.Bool
+ first := qr.scan.pf.items[0].bc
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ if bc == first {
+ return decodeCursor(ctx, bc, desc)
+ }
+ running.Add(1)
+ <-gate
+ if released.Load() {
+ panic("cursor released while its decode was running")
+ }
+ return decodeCursor(ctx, bc, desc)
+ }
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ b.Release()
+ require.Eventually(t, func() bool { return running.Load() > 0 }, 5*time.Second, time.Millisecond)
+ // The next cursor the scan takes is requested and running; cancel the pull.
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ returned := make(chan error, 1)
+ go func() {
+ _, pullErr := qr.PullBatch(ctx)
+ returned <- pullErr
+ }()
+ select {
+ case <-returned:
+ released.Store(true)
+ t.Fatal("PullBatch returned, and released the cursor, while its decode was running")
+ case <-time.After(100 * time.Millisecond):
+ }
+ close(gate)
+ select {
+ case pullErr := <-returned:
+ require.ErrorIs(t, pullErr, context.Canceled)
+ case <-time.After(5 * time.Second):
+ t.Fatal("PullBatch did not return after the decode finished")
+ }
+ qr.Release()
+ require.Zero(t, budget.held.Load())
+ require.Equal(t, before, blockCursorPool.RefsCount())
+ pool.close()
+ waitWorkers(t, sharedWorkers)
+}
+
+func TestScanReportsAContextCanceledWhileWaitingForADecode(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 10, 100)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ pool := newDecodePool(1, nil)
+ defer pool.close()
+ before := blockCursorPool.RefsCount()
+
+ ctx, cancel := context.WithCancel(context.Background())
+ qr := fx.newResult(t, q, schema)
+ qr.dp, qr.ctx = pool, ctx
+ require.True(t, qr.useWindowScan())
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ cancel()
+ return decodeCursor(ctx, bc, desc)
+ }
+ _, err := qr.PullBatch(ctx)
+ require.Error(t, err)
+ require.True(t, errors.Is(err, context.Canceled), "got %v", err)
+ require.Contains(t, err.Error(), "interrupt")
+ qr.Release()
+ require.Equal(t, before, blockCursorPool.RefsCount())
+}
+
+// awaitTask waits for a task that was queued on a pool directly.
+func awaitTask(t *decodeTask) {
+ for !t.finished() {
+ time.Sleep(time.Millisecond)
+ }
+}
+
+func TestDecodePoolCloseCancelsQueuedTasksAndWaitsForRunningOnes(t *testing.T) {
+ verifyNoScanLeaks(t)
+ onlyTheSharedPool(t)
+ pool := newDecodePool(1, nil)
+ gate := make(chan struct{})
+ started := make(chan struct{}, 1)
+ slow := func(ctx context.Context, _ *blockCursor, _ bool) (bool, error) {
+ started <- struct{}{}
+ <-gate
+ return true, ctx.Err()
+ }
+ cursors := stubCursors(3, 1)
+ defer releaseCursors(cursors)
+ tasks := make([]*decodeTask, len(cursors))
+ wake := make(chan struct{}, 1)
+ for i, bc := range cursors {
+ tasks[i] = &decodeTask{}
+ tasks[i].init(context.Background(), slow, bc, false, nil, wake)
+ pool.submit(tasks[i], false)
+ }
+ <-started
+ closed := make(chan struct{})
+ go func() {
+ pool.close()
+ close(closed)
+ }()
+ select {
+ case <-closed:
+ t.Fatal("close returned while a decode was running")
+ case <-time.After(50 * time.Millisecond):
+ }
+ close(gate)
+ <-closed
+ pool.close() // idempotent
+ awaitTask(tasks[0])
+ require.NoError(t, tasks[0].err)
+ for _, tk := range tasks[1:] {
+ awaitTask(tk)
+ require.ErrorIs(t, tk.err, errDecodePoolClosed)
+ }
+ late := &decodeTask{}
+ late.init(context.Background(), slow, cursors[0], false, nil, wake)
+ pool.submit(late, false)
+ awaitTask(late)
+ require.ErrorIs(t, late.err, errDecodePoolClosed, "a closed pool cancels instead of queueing")
+ waitWorkers(t, sharedWorkers)
+}
+
+func TestDecodePoolRecoversFromAPanicInTheLoader(t *testing.T) {
+ verifyNoScanLeaks(t)
+ pool := newDecodePool(1, nil)
+ defer pool.close()
+ cursors := stubCursors(2, 1)
+ defer releaseCursors(cursors)
+ var stats scanStats
+ n := 0
+ load := cursorLoader(func(_ context.Context, _ *blockCursor, _ bool) (bool, error) {
+ n++
+ if n == 1 {
+ panic("boom")
+ }
+ return true, nil
+ })
+ pf := newPrefetcher(context.Background(), pool, newScanBudget(-1, nil), nil, &stats, &load, false, cursors)
+ pf.pump()
+ ok, err := pf.take(context.Background(), cursors[0])
+ require.False(t, ok)
+ require.ErrorContains(t, err, "boom")
+ ok, err = pf.take(context.Background(), cursors[1])
+ require.NoError(t, err)
+ require.True(t, ok, "the worker survived")
+ pf.close()
+}
+
+func TestPrefetchPropagatesLoaderErrorsAndFreesTheReservation(t *testing.T) {
+ verifyNoScanLeaks(t)
+ pool := newDecodePool(2, nil)
+ defer pool.close()
+ cursors := stubCursors(4, 50)
+ var stats scanStats
+ boom := errors.New("disk read failed")
+ load := cursorLoader(func(_ context.Context, bc *blockCursor, _ bool) (bool, error) {
+ if bc.bm.seriesID == 2 {
+ return false, boom
+ }
+ return true, nil
+ })
+ budget := newScanBudget(1000, pool)
+ pf := newPrefetcher(context.Background(), pool, budget, nil, &stats, &load, false, cursors)
+ pf.pump()
+ for i, bc := range cursors[:2] {
+ ok, err := pf.take(context.Background(), bc)
+ if i == 1 {
+ require.ErrorIs(t, err, boom)
+ require.False(t, ok)
+ } else {
+ require.NoError(t, err)
+ }
+ }
+ pf.close()
+ releaseCursors(cursors)
+ require.Zero(t, budget.held.Load(), "releasing the cursors, decoded or not, frees every reservation")
+ require.Zero(t, pool.reserved.Load())
+}
+
+// fakeMemory is a protector whose headroom a test controls.
+type fakeMemory struct {
+ protector.Nop
+ avail atomic.Int64
+}
+
+func (f *fakeMemory) AvailableBytes() int64 { return f.avail.Load() }
+
+func TestNodeHeadroomGatesPrefetchButNeverTheCursorTheScanNeeds(t *testing.T) {
+ verifyNoScanLeaks(t)
+ mem := &fakeMemory{}
+ mem.avail.Store(250)
+ pool := newDecodePool(2, mem)
+ defer pool.close()
+ b1 := newScanBudget(-1, pool)
+ b2 := newScanBudget(-1, pool)
+
+ var r1, r2, r3 scanReservation
+ require.Equal(t, reserveOK, b1.tryReserve(100, true, &r1))
+ require.Equal(t, reserveOK, b2.tryReserve(100, true, &r2))
+ require.Equal(t, reserveOverNode, b2.tryReserve(100, true, &r3), "300 would pass the 250 bytes free: prefetch waits")
+ require.EqualValues(t, 200, pool.reserved.Load(), "a refused reservation leaves no trace")
+ require.Zero(t, b2.held.Load()-100)
+
+ mem.avail.Store(1 << 20)
+ require.NoError(t, b2.reserveCurrent(context.Background(), 100, &r3), "the current cursor is not refused")
+ r1.release()
+ r2.release()
+ r3.release()
+ r3.release() // idempotent
+ require.Zero(t, pool.reserved.Load())
+ require.Zero(t, b1.held.Load()+b2.held.Load())
+
+ // With no memory left the scan still takes the reservation it needs; the
+ // protector, not the counter, decides how long it waits (AcquireResource).
+ mem.avail.Store(0)
+ var r4 scanReservation
+ require.Equal(t, reserveOK, b1.tryReserve(100, false, &r4))
+ require.EqualValues(t, 100, pool.reserved.Load())
+ r4.release()
+ require.Zero(t, pool.reserved.Load())
+}
+
+// holdingMemory is a protector that never admits: AcquireResource waits for ctx.
+type holdingMemory struct {
+ protector.Nop
+ acquires atomic.Int32
+}
+
+func (*holdingMemory) AvailableBytes() int64 { return 1 << 30 }
+
+func (h *holdingMemory) AcquireResource(ctx context.Context, _ uint64) error {
+ h.acquires.Add(1)
+ <-ctx.Done()
+ return ctx.Err()
+}
+
+func TestReserveCurrentWaitsOnTheProtectorOnlyWhileTheQueryHoldsNothing(t *testing.T) {
+ verifyNoScanLeaks(t)
+ mem := &holdingMemory{}
+ pool := newDecodePool(1, mem)
+ defer pool.close()
+ b := newScanBudget(-1, pool)
+
+ // Holding nothing: the query is being admitted, and waits for the protector.
+ ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+ defer cancel()
+ var r1 scanReservation
+ require.ErrorIs(t, b.reserveCurrent(ctx, 100, &r1), context.DeadlineExceeded)
+ require.Zero(t, b.held.Load(), "a refused admission holds nothing")
+
+ // Holding memory: the current cursor must never block on the protector.
+ var r2, r3 scanReservation
+ require.Equal(t, reserveOK, b.tryReserve(100, false, &r2))
+ acquires := mem.acquires.Load()
+ done := make(chan error, 1)
+ go func() { done <- b.reserveCurrent(context.Background(), 100, &r3) }()
+ select {
+ case err := <-done:
+ require.NoError(t, err)
+ case <-time.After(5 * time.Second):
+ t.Fatal("reserveCurrent blocked on the protector while the query held memory")
+ }
+ require.Equal(t, acquires, mem.acquires.Load())
+ r2.release()
+ r3.release()
+}
+
+func TestTwoQueriesHoldingFrontiersBeyondTheHeadroomBothComplete(t *testing.T) {
+ verifyNoScanLeaks(t)
+ mem := &holdingMemory{}
+ pool := newDecodePool(2, mem)
+ defer pool.close()
+ b1, b2 := newScanBudget(-1, pool), newScanBudget(-1, pool)
+ // Each query is admitted (held == 0 at its first reservation only because the
+ // fake admits through a context that ends at once), then holds a frontier.
+ var first [2]scanReservation
+ require.Equal(t, reserveOK, b1.tryReserve(100, false, &first[0]))
+ require.Equal(t, reserveOK, b2.tryReserve(100, false, &first[1]))
+ ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+ defer cancel()
+ var more [2][4]scanReservation
+ var wg sync.WaitGroup
+ errs := make([]error, 2)
+ for i, b := range []*scanBudget{b1, b2} {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ for k := range more[i] {
+ if errs[i] = b.reserveCurrent(ctx, 100, &more[i][k]); errs[i] != nil {
+ return
+ }
+ }
+ }()
+ }
+ wg.Wait()
+ require.NoError(t, errs[0])
+ require.NoError(t, errs[1])
+ require.EqualValues(t, 1000, pool.reserved.Load(), "beyond the headroom, yet nothing hung")
+ for i := range more {
+ first[i].release()
+ for k := range more[i] {
+ more[i][k].release()
+ }
+ }
+ require.Zero(t, pool.reserved.Load())
+}
+
+func TestTakePromotesAQueuedPrefetchAheadOfTheNormalQueue(t *testing.T) {
+ verifyNoScanLeaks(t)
+ onlyTheSharedPool(t)
+ pool := newDecodePool(1, nil)
+ defer pool.close()
+ cursors := stubCursors(4, 1)
+ defer releaseCursors(cursors)
+ blocker, foreign := generateBlockCursor(), generateBlockCursor()
+ defer releaseBlockCursor(blocker)
+ defer releaseBlockCursor(foreign)
+ gate := make(chan struct{})
+ started := make(chan *blockCursor, 16)
+ var load cursorLoader = func(_ context.Context, bc *blockCursor, _ bool) (bool, error) {
+ started <- bc
+ if bc == blocker {
+ <-gate
+ }
+ return true, nil
+ }
+ wake := make(chan struct{}, 8)
+ var tb, tf decodeTask
+ tb.init(context.Background(), load, blocker, false, nil, wake)
+ pool.submit(&tb, false)
+ require.Same(t, blocker, <-started, "the only worker is busy")
+
+ // The foreign read-ahead is queued before the scan's, so it is served first
+ // unless the scan's current cursor is promoted.
+ tf.init(context.Background(), load, foreign, false, nil, wake)
+ pool.submit(&tf, false)
+ pf := newPrefetcher(context.Background(), pool, nil, nil, &scanStats{}, &load, false, cursors)
+ pf.pump()
+ require.True(t, pf.items[0].requested)
+
+ done := make(chan struct{})
+ go func() {
+ ok, err := pf.take(context.Background(), cursors[0])
+ require.NoError(t, err)
+ require.True(t, ok)
+ close(done)
+ }()
+ require.Eventually(t, func() bool {
+ pool.mu.Lock()
+ defer pool.mu.Unlock()
+ return len(pool.urgent) == 1
+ }, 5*time.Second, time.Millisecond)
+ close(gate)
+ <-done
+ require.Same(t, cursors[0], <-started, "the promoted cursor runs before the older normal entry")
+ require.Same(t, foreign, <-started)
+ awaitTask(&tf)
+ pf.close()
+ // The stale normal-queue entry is a no-op: the promoted cursor decoded once.
+ pool.close()
+ for len(started) > 0 {
+ require.NotSame(t, cursors[0], <-started)
+ }
+}
+
+func TestScanSequenceMatchesTheOrderCursorsAreDecoded(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fixtures := []struct {
+ fx *fxFixture
+ q *fxQuery
+ }{
+ {mixedFixture(t), prefetchQuery(0, 30000, false, true)},
+ {mixedFixture(t), prefetchQuery(0, 30000, true, true)},
+ {mixedFixture(t), prefetchQuery(0, 30000, true, false)},
+ {oapFixture(t, 16), prefetchQuery(0, 1<<20, true, true)},
+ {oapFixture(t, 16), prefetchQuery(0, 1<<20, true, false)},
+ {staggeredFixture(t, 9), prefetchQuery(0, 1<<20, true, false)},
+ {staggeredFixture(t, 9), prefetchQuery(100, 500, true, true)},
+ }
+ type key struct {
+ part uint64
+ sid common.SeriesID
+ ts int64
+ }
+ keyOf := func(bc *blockCursor) key { return key{bc.p.partMetadata.ID, bc.bm.seriesID, bc.bm.timestamps.min} }
+ for i, tt := range fixtures {
+ t.Run(fmt.Sprintf("query %d", i), func(t *testing.T) {
+ qr := tt.fx.newResult(t, tt.q, tt.q.schema(t))
+ qr.dp = nil
+ require.True(t, qr.useWindowScan())
+ var want, got []key
+ for _, bc := range qr.scan.sequence() {
+ want = append(want, keyOf(bc))
+ }
+ qr.scan.load = func(ctx context.Context, bc *blockCursor, desc bool) (bool, error) {
+ got = append(got, keyOf(bc))
+ return decodeCursor(ctx, bc, desc)
+ }
+ drainRows(t, qr)
+ qr.Release()
+ require.Equal(t, want, got, "the prefetch sequence is the order the scan decodes in")
+ })
+ }
+}
+
+// TestConcurrentQueriesShareTheNodePool runs many queries, some canceled and some
+// closed early, over one pool and one node budget under the race detector.
+func TestConcurrentQueriesShareTheNodePool(t *testing.T) {
+ verifyNoScanLeaks(t)
+ mixed := mixedFixture(t)
+ oapSeries, rounds := 20, 6
+ if raceEnabled {
+ oapSeries, rounds = 12, 3
+ }
+ oap := oapFixture(t, oapSeries)
+ queries := []struct {
+ fx *fxFixture
+ q *fxQuery
+ }{
+ {mixed, prefetchQuery(0, 30000, false, true)},
+ {mixed, prefetchQuery(0, 30000, true, false)},
+ {oap, prefetchQuery(0, 1<<20, true, true)},
+ {oap, prefetchQuery(0, 1<<20, false, true)},
+ }
+ want := make([][]string, len(queries))
+ for i, tt := range queries {
+ qr := tt.fx.newResult(t, tt.q, tt.q.schema(t))
+ qr.dp = nil
+ want[i] = drainRows(t, qr)
+ qr.Release()
+ }
+ before := blockCursorPool.RefsCount()
+ onlyTheSharedPool(t)
+ mem := &fakeMemory{}
+ mem.avail.Store(-1)
+ pool := newDecodePool(4, mem)
+
+ var wg sync.WaitGroup
+ var budgets []*scanBudget
+ var bmu sync.Mutex
+ for w := range 24 {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ rng := rand.New(rand.NewSource(int64(w))) //nolint:gosec // deterministic test workload
+ for round := range rounds {
+ i := (w + round) % len(queries)
+ tt := queries[i]
+ qr := tt.fx.newResult(t, tt.q, tt.q.schema(t))
+ ctx, cancel := context.WithCancel(context.Background())
+ qr.ctx = ctx
+ budget := newScanBudget(maxBlockBytes(qr)*int64(3+rng.Intn(20)), pool)
+ bmu.Lock()
+ budgets = append(budgets, budget)
+ bmu.Unlock()
+ qr.dp, qr.budget = pool, budget
+ switch rng.Intn(3) {
+ case 0: // full drain, which must match
+ got, err := drainRowsErr(qr)
+ switch {
+ case err != nil && !strings.Contains(err.Error(), "block scan quota exceeded"):
+ t.Errorf("worker %d: %v", w, err)
+ case err == nil && !slices.Equal(want[i], got):
+ t.Errorf("worker %d: rows differ from the synchronous scan", w)
+ }
+ case 1: // early close after a few batches, as a LIMIT does
+ for range 1 + rng.Intn(3) {
+ b, err := qr.PullBatch(ctx)
+ if err != nil || b == nil {
+ break
+ }
+ b.Release()
+ }
+ case 2: // cancel in the middle
+ b, err := qr.PullBatch(ctx)
+ if err == nil && b != nil {
+ b.Release()
+ }
+ cancel()
+ for {
+ b, err = qr.PullBatch(ctx)
+ if err != nil || b == nil {
+ break
+ }
+ b.Release()
+ }
+ }
+ qr.Release()
+ cancel()
+ }
+ }()
+ }
+ wg.Wait()
+ for _, b := range budgets {
+ require.Zero(t, b.held.Load())
+ }
+ require.Zero(t, pool.reserved.Load())
+ require.Equal(t, before, blockCursorPool.RefsCount())
+ pool.close()
+ waitWorkers(t, sharedWorkers)
+ waitNoDecodes(t)
+}
+
+// TestConcurrentQueriesStayIdenticalToTheSynchronousScan checks the rows of full
+// drains that share the pool with other queries.
+func TestConcurrentQueriesStayIdenticalToTheSynchronousScan(t *testing.T) {
+ verifyNoScanLeaks(t)
+ oap := oapFixture(t, 20)
+ q := prefetchQuery(0, 1<<20, true, true)
+ schema := q.schema(t)
+ sync0 := oap.newResult(t, q, schema)
+ sync0.dp = nil
+ want := drainRows(t, sync0)
+ sync0.Release()
+ pool := newDecodePool(3, nil)
+ defer pool.close()
+ var wg sync.WaitGroup
+ for range 12 {
+ wg.Add(1)
+ go func() {
+ defer wg.Done()
+ qr := oap.newResult(t, q, schema)
+ qr.dp = pool
+ got := drainRows(t, qr)
+ qr.Release()
+ if !slices.Equal(want, got) {
+ t.Errorf("rows differ from the synchronous scan")
+ }
+ }()
+ }
+ wg.Wait()
+}
+
+func TestWholeResultAdmissionStillGuardsThePathsThatLoadEverything(t *testing.T) {
+ verifyNoScanLeaks(t)
+ fx := manySeries(t, 4, 100)
+ q := prefetchQuery(0, 1000, false, true)
+ schema := q.schema(t)
+ quotaErr := errors.New("block scan quota exceeded: used 9 bytes, quota is 1 bytes")
+
+ qr := fx.newResult(t, q, schema)
+ qr.admit = func(context.Context) error { return quotaErr }
+ r := qr.Pull()
+ require.NotNil(t, r)
+ require.ErrorIs(t, r.Error, quotaErr)
+ qr.Release()
+
+ defer setLegacyBatchScan(setLegacyBatchScan(true))
+ qr = fx.newResult(t, q, schema)
+ qr.admit = func(context.Context) error { return quotaErr }
+ _, err := qr.PullBatch(context.Background())
+ require.ErrorIs(t, err, quotaErr)
+ qr.Release()
+
+ calls := 0
+ qr = fx.newResult(t, q, schema)
+ qr.admit = func(context.Context) error { calls++; return nil }
+ require.Len(t, drainRows(t, qr), 400)
+ qr.Release()
+ require.Equal(t, 1, calls, "admission runs once")
+}
+
+// TestPrefetchAddsAtMostAQuarterOfTheFrontierToTheMergePeak checks, through the
+// reservations (the decoded memory a scan holds), that decoding ahead keeps a merge
+// within the frontier it already needed plus the prefetch allowance.
+func TestPrefetchAddsAtMostAQuarterOfTheFrontierToTheMergePeak(t *testing.T) {
+ verifyNoScanLeaks(t)
+ const (
+ series = frontierSeries
+ blocks = 12
+ span = 200
+ )
+ parts := make([]fxPart, blocks)
+ for p := range parts {
+ for sid := common.SeriesID(1); sid <= series; sid++ {
+ parts[p] = append(parts[p], fxRange{sid, int64(p*span + 1), int64((p + 1) * span), int64(p + 1)})
+ }
+ }
+ fx := newFixture(t, parts...)
+ pool := newDecodePool(4, nil)
+ defer pool.close()
+ for _, asc := range []bool{true, false} {
+ q := prefetchQuery(0, 1<<20, true, asc)
+ schema := q.schema(t)
+ qr := fx.newResult(t, q, schema)
+ block := maxBlockBytes(qr)
+ budget := newScanBudget(1<<30, pool)
+ qr.dp, qr.budget = pool, budget
+ var peakHeld int64
+ rows := 0
+ for {
+ b, err := qr.PullBatch(context.Background())
+ require.NoError(t, err)
+ if b == nil {
+ break
+ }
+ rows += b.RowCount()
+ b.Release()
+ peakHeld = max(peakHeld, budget.held.Load())
+ }
+ require.Equal(t, series*blocks*span, rows)
+ bound := (series + series/4 + minFinishedLimit + max(prefetchMaxAhead, series/4)) * block
+ require.LessOrEqual(t, peakHeld, bound, "asc=%v: reservations held between pulls", asc)
+ require.Greater(t, peakHeld, int64(series)*block/2)
+ qr.Release()
+ require.Positive(t, qr.scan.stats.prefetchPeak, "the peak is recorded when the scan closes")
+ var allowedAhead int64 = max(prefetchMaxAhead, series/4)
+ require.LessOrEqual(t, qr.scan.stats.prefetchPeak, allowedAhead)
+ require.Zero(t, budget.held.Load())
+ }
+}
diff --git a/banyand/measure/scan_resources.go b/banyand/measure/scan_resources.go
new file mode 100644
index 000000000..dc039a925
--- /dev/null
+++ b/banyand/measure/scan_resources.go
@@ -0,0 +1,83 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "sync/atomic"
+
+ "github.com/apache/skywalking-banyandb/pkg/pool"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+// Process-wide counts of what the vectorized scan holds (measure scan path v2,
+// step 4d). They exist so that tests and the soak can prove the scan gives
+// everything back: each is one atomic add on a path that already does an atomic
+// add of its own.
+var (
+ liveReservedBytes atomic.Int64
+ liveDecodeTasks atomic.Int64
+ liveDecodeWorkers atomic.Int64
+ queriesStarted atomic.Int64
+)
+
+const (
+ cursorPoolName = "measure-blockCursor"
+ queryResultPoolKey = "measure.queryResult"
+)
+
+// ScanResources is a snapshot of the resources the measure scan currently holds.
+type ScanResources struct {
+ // CursorRefs is the number of block cursors checked out of the cursor pool: held
+ // by a scan or pinned by a batch that borrows their buffers.
+ CursorRefs int
+ // OwnerPins is the number of owner references held by batches that borrow
+ // decoded memory.
+ OwnerPins int64
+ // QueryResults is the number of query results not yet released.
+ QueryResults int
+ // ReservedBytes is the total scan memory reservations across every query.
+ ReservedBytes int64
+ // DecodeTasks is the number of decode tasks queued or running on a decode pool.
+ DecodeTasks int64
+ // DecodeWorkers is the number of decode pool worker goroutines running.
+ DecodeWorkers int64
+}
+
+// ScanResourceSnapshot returns the resources the scan holds right now. The cursor
+// and query-result counts are zero in slim builds, where the pools do not count.
+func ScanResourceSnapshot() ScanResources {
+ refs := pool.AllRefsCount()
+ return ScanResources{
+ CursorRefs: refs[cursorPoolName],
+ OwnerPins: vectorized.OutstandingOwnerPins(),
+ QueryResults: refs[queryResultPoolKey],
+ ReservedBytes: liveReservedBytes.Load(),
+ DecodeTasks: liveDecodeTasks.Load(),
+ DecodeWorkers: liveDecodeWorkers.Load(),
+ }
+}
+
+// Idle reports whether the scan holds nothing. Decode workers are not counted: a
+// pool's workers live as long as the pool.
+func (r ScanResources) Idle() bool {
+ return r.CursorRefs == 0 && r.OwnerPins == 0 && r.QueryResults == 0 && r.ReservedBytes == 0 && r.DecodeTasks == 0
+}
+
+// QueriesStarted returns how many measure queries this process has started. It never
+// decreases, so a soak can check that queries keep flowing.
+func QueriesStarted() int64 { return queriesStarted.Load() }
diff --git a/banyand/measure/scan_window.go b/banyand/measure/scan_window.go
new file mode 100644
index 000000000..13a41af3b
--- /dev/null
+++ b/banyand/measure/scan_window.go
@@ -0,0 +1,642 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "math"
+ "sort"
+ "sync/atomic"
+
+ "github.com/pkg/errors"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+)
+
+// Series-window scan (measure scan path v2, steps 4a and 4c).
+//
+// The matching blocks of a query are grouped into series windows: all the
+// in-range blocks of one series, from every part. The windows are cut into units
+// that are visited one after another in the order the legacy heap merge would have
+// produced their rows:
+//
+// - Without a time order, a unit is one window, and windows follow the series
+// order of the query.
+// - With a time order, windows whose time ranges are disjoint from every other
+// window are units of their own, in time order. Windows whose ranges overlap
+// or touch interleave their rows, so they form one merge unit (scan_merge.go).
+//
+// A window is cut into segments from block metadata alone: blocks ordered by their
+// first timestamp, a block that starts at or before the previous segment's last
+// timestamp (an equal boundary may be a duplicate version) joining that segment. A
+// segment is decoded only when the scan reaches it, so decoded memory is bounded
+// by one segment instead of by the whole result.
+//
+// - A segment of one block is emitted as a batch that lends the block's decoded
+// columns (blockCursor.lendBatch); nothing is copied.
+// - A segment of several overlapping blocks, and a merge unit, go through
+// timeMerge: keys first, then one gather per column, decoding each block only
+// when the merge reaches it.
+//
+// Decoding goes through the scan's prefetcher (scan_prefetch.go): the plan (units,
+// windows and their segments) is known up front, so the node-wide decode pool of step
+// 4b decodes the next cursors while the current one is emitted, in the order the scan
+// will take them, each under a reservation released with the cursor.
+
+// coalesceTargetRows is how many rows the scan gathers into one owned batch from
+// segments smaller than model.SmallBatchRows before emitting it.
+const coalesceTargetRows = 1024
+
+// maxSelectionRows is the largest batch a uint16 Selection can address.
+const maxSelectionRows = math.MaxUint16 + 1
+
+// legacyBatchScan routes PullBatch to the pre-window implementation (decode every
+// block, then merge row by row). It exists so tests can compare the two.
+var legacyBatchScan atomic.Bool
+
+// setLegacyBatchScan switches the legacy scan and returns the previous setting.
+func setLegacyBatchScan(on bool) bool { return legacyBatchScan.Swap(on) }
+
+// scanStats counts what a window scan did. It is read by tests and mirrored to
+// the query metrics.
+type scanStats struct {
+ // windows is the number of series windows started.
+ windows int64
+ // windowsOverlapped is the number of windows with at least one segment of two
+ // or more blocks, decided from metadata.
+ windowsOverlapped int64
+ // segmentsFast is the number of segments emitted by lending one block.
+ segmentsFast int64
+ // segmentsOverlap is the number of segments of two or more blocks, decided from
+ // metadata, that took the merge.
+ segmentsOverlap int64
+ // coalesced is the number of batches emitted by copying segments smaller
+ // than model.SmallBatchRows together.
+ coalesced int64
+ // mergeUnits is the number of merges started: one per overlapping segment and
+ // one per group of series whose time ranges overlap.
+ mergeUnits int64
+ // mergePages is the number of batches the merges emitted.
+ mergePages int64
+ // mergeRows is the number of rows the merges emitted.
+ mergeRows int64
+ // mergeCursors is the number of cursors the merges decoded, each when the merge
+ // frontier reached it.
+ mergeCursors int64
+ // mergePeakLive is the most cursors any one merge held decoded at once.
+ mergePeakLive int64
+ // prefetchPeak is the most decodes the scan had requested ahead of the one it
+ // was consuming.
+ prefetchPeak int64
+ // reservationWaits is the number of times prefetch found no room in the budget
+ // and waited; reservationFails the number of times the cursor the scan needed
+ // could not be reserved.
+ reservationWaits int64
+ reservationFails int64
+}
+
+// scanSegment is a run of blocks of one series whose time ranges chain together.
+type scanSegment struct {
+ cursors []*blockCursor
+}
+
+// seriesWindow is every in-range block of one series, cut into segments in
+// emission order.
+type seriesWindow struct {
+ // cursors is every cursor of the series in arrival order; planning cuts it
+ // into segments and the segments take over the cursors.
+ cursors []*blockCursor
+ segments []scanSegment
+ sid common.SeriesID
+ minTS int64
+ maxTS int64
+}
+
+// scanUnit is what the scan visits as one step: a single window, or a group of
+// windows whose rows interleave in time and are merged together.
+type scanUnit struct {
+ // windows holds one window for a window unit and two or more for a merge unit.
+ windows []seriesWindow
+}
+
+// windowScan streams the units of one queryResult.
+type windowScan struct {
+ qr *queryResult
+ cur *seriesWindow
+ merge *timeMerge
+ load cursorLoader
+ pf *prefetcher
+ spare *timeMerge
+ plan []colPlan
+ builder *model.MeasureBatch
+ ready *model.MeasureBatch
+ err error
+ units []scanUnit
+ boundary []int
+ stats scanStats
+ unitIdx int
+ segIdx int
+ desc bool
+ started bool
+}
+
+// newWindowScan plans the scan over qr.data and takes ownership of its cursors.
+func newWindowScan(qr *queryResult) *windowScan {
+ s := &windowScan{qr: qr, desc: qr.orderByTimestampDesc(), load: decodeCursor}
+ s.units = planUnits(qr, groupWindows(qr.data), s.desc)
+ for i := range qr.data {
+ qr.data[i] = nil
+ }
+ qr.data = qr.data[:0]
+ for i := range s.units {
+ if len(s.units[i].windows) != 1 {
+ continue
+ }
+ w := &s.units[i].windows[0]
+ w.segments = planSegments(w.cursors, s.desc)
+ w.cursors = nil
+ }
+ s.pf = newPrefetcher(qr.ctx, qr.dp, qr.budget, qr.qm, &s.stats, &s.load, s.desc, s.sequence())
+ return s
+}
+
+// sequence lists every cursor of the plan in the order the scan decodes them.
+func (s *windowScan) sequence() []*blockCursor {
+ var seq []*blockCursor
+ for i := range s.units {
+ unit := &s.units[i]
+ if len(unit.windows) == 1 {
+ for _, seg := range unit.windows[0].segments {
+ if len(seg.cursors) == 1 {
+ seq = append(seq, seg.cursors[0])
+ continue
+ }
+ seq = append(seq, orderPending(seg.cursors, s.desc)...)
+ }
+ continue
+ }
+ var all []*blockCursor
+ for j := range unit.windows {
+ all = append(all, unit.windows[j].cursors...)
+ }
+ seq = append(seq, orderPending(all, s.desc)...)
+ }
+ return seq
+}
+
+// loadCursor is the cursorLoader of the scan's merges: it takes the next decode of
+// the sequence.
+func (s *windowScan) loadCursor(ctx context.Context, bc *blockCursor, _ bool) (bool, error) {
+ return s.pf.take(ctx, bc)
+}
+
+// groupWindows groups cursors by series, keeping first-seen series order and the
+// order the cursors arrived in within a series. The windows hold the cursors
+// uncut; planSegments cuts them.
+func groupWindows(data []*blockCursor) []seriesWindow {
+ index := make(map[common.SeriesID]int)
+ var windows []seriesWindow
+ for _, bc := range data {
+ i, ok := index[bc.bm.seriesID]
+ if !ok {
+ i = len(windows)
+ index[bc.bm.seriesID] = i
+ windows = append(windows, seriesWindow{
+ sid: bc.bm.seriesID,
+ minTS: bc.bm.timestamps.min,
+ maxTS: bc.bm.timestamps.max,
+ })
+ }
+ w := &windows[i]
+ w.cursors = append(w.cursors, bc)
+ w.minTS = min(w.minTS, bc.bm.timestamps.min)
+ w.maxTS = max(w.maxTS, bc.bm.timestamps.max)
+ }
+ return windows
+}
+
+// planUnits orders windows into the units the scan visits, in the order the legacy
+// heap yields their rows.
+//
+// Without a time order the heap sorts by the series' position in the query, then
+// by time, so every window is a unit and units follow sidToIndex. With a time order
+// the heap interleaves series by timestamp, so windows are swept in time (latest
+// first for a descending order) and a window joins the unit before it when its
+// range reaches the unit's range (an equal timestamp counts: there the series are
+// ordered by id). A unit of one window is emitted whole; a unit of several is
+// merged.
+func planUnits(qr *queryResult, windows []seriesWindow, desc bool) []scanUnit {
+ if !qr.orderByTS {
+ sort.SliceStable(windows, func(i, j int) bool {
+ return qr.sidToIndex[windows[i].sid] < qr.sidToIndex[windows[j].sid]
+ })
+ units := make([]scanUnit, len(windows))
+ for i := range windows {
+ units[i].windows = windows[i : i+1 : i+1]
+ }
+ return units
+ }
+ if desc {
+ sort.SliceStable(windows, func(i, j int) bool { return windows[i].maxTS > windows[j].maxTS })
+ } else {
+ sort.SliceStable(windows, func(i, j int) bool { return windows[i].minTS < windows[j].minTS })
+ }
+ var units []scanUnit
+ start := 0
+ var edge int64 // the unit's last timestamp (first, descending) reached so far
+ for i := range windows {
+ w := &windows[i]
+ if i > start {
+ if (!desc && w.minTS <= edge) || (desc && w.maxTS >= edge) {
+ if desc {
+ edge = min(edge, w.minTS)
+ } else {
+ edge = max(edge, w.maxTS)
+ }
+ continue
+ }
+ units = append(units, scanUnit{windows: windows[start:i:i]})
+ start = i
+ }
+ if desc {
+ edge = w.minTS
+ } else {
+ edge = w.maxTS
+ }
+ }
+ if len(windows) > 0 {
+ units = append(units, scanUnit{windows: windows[start:]})
+ }
+ return units
+}
+
+// planSegments cuts one series' cursors into segments from block metadata. The
+// cursors are ordered by their first timestamp (arrival order breaks ties); a
+// cursor whose first timestamp is at or before the running last timestamp of the
+// segment before it joins that segment. For a descending scan the segments come
+// out latest first.
+func planSegments(cursors []*blockCursor, desc bool) []scanSegment {
+ if len(cursors) == 1 {
+ return []scanSegment{{cursors: cursors}}
+ }
+ sorted := make([]*blockCursor, len(cursors))
+ copy(sorted, cursors)
+ sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].bm.timestamps.min < sorted[j].bm.timestamps.min })
+ var segments []scanSegment
+ var segMax int64
+ for _, bc := range sorted {
+ if len(segments) > 0 && bc.bm.timestamps.min <= segMax {
+ last := &segments[len(segments)-1]
+ last.cursors = append(last.cursors, bc)
+ segMax = max(segMax, bc.bm.timestamps.max)
+ continue
+ }
+ segments = append(segments, scanSegment{cursors: []*blockCursor{bc}})
+ segMax = bc.bm.timestamps.max
+ }
+ if desc {
+ for i, j := 0, len(segments)-1; i < j; i, j = i+1, j-1 {
+ segments[i], segments[j] = segments[j], segments[i]
+ }
+ }
+ return segments
+}
+
+// next returns the next batch, or nil at the end of the scan. The error is
+// sticky. Every batch is marked with what the scan knows about its rows: where each
+// series ends, and that time runs in the scan's direction within a series (a series'
+// segments are visited in time order, and a merge emits its rows in time order), so
+// an aggregation can split the batch into runs without reading its rows.
+func (s *windowScan) next(ctx context.Context) (*model.MeasureBatch, error) {
+ b, err := s.nextUnmarked(ctx)
+ if b != nil {
+ b.SeriesKnown = true
+ b.TimeOrder = vectorized.TimeOrderAsc
+ if s.desc {
+ b.TimeOrder = vectorized.TimeOrderDesc
+ }
+ }
+ return b, err
+}
+
+func (s *windowScan) nextUnmarked(ctx context.Context) (*model.MeasureBatch, error) {
+ if s.err != nil {
+ return nil, s.err
+ }
+ if !s.started {
+ s.started = true
+ s.pf.pump()
+ }
+ for {
+ if s.ready != nil {
+ b := s.ready
+ s.ready = nil
+ return b, nil
+ }
+ if s.builder != nil && s.pf.waiting() {
+ // The batch being coalesced pins the cursors of the pieces it holds, and the
+ // budget has no room for more: hand it over now rather than wait to fill it.
+ return s.takeBuilder(), nil
+ }
+ piece, err := s.nextPiece(ctx)
+ if err != nil {
+ s.err = err
+ return nil, err
+ }
+ if piece == nil {
+ if s.builder != nil {
+ return s.takeBuilder(), nil
+ }
+ return nil, nil
+ }
+ if piece.ActiveRows() < model.SmallBatchRows {
+ s.appendToBuilder(piece)
+ if s.builder.RowCount() >= coalesceTargetRows {
+ return s.takeBuilder(), nil
+ }
+ continue
+ }
+ if s.builder != nil {
+ s.ready = piece
+ return s.takeBuilder(), nil
+ }
+ return piece, nil
+ }
+}
+
+// nextPiece returns the next batch of one unit: a lent block, or one page of a
+// merge. nil means the scan is exhausted.
+func (s *windowScan) nextPiece(ctx context.Context) (*model.MeasureBatch, error) {
+ for {
+ if s.merge != nil {
+ b, err := s.merge.nextPage(ctx)
+ if err != nil {
+ return nil, err
+ }
+ if b != nil {
+ return b, nil
+ }
+ s.finishMerge()
+ continue
+ }
+ if s.cur == nil || s.segIdx >= len(s.cur.segments) {
+ if s.unitIdx >= len(s.units) {
+ s.cur = nil
+ return nil, nil
+ }
+ unit := &s.units[s.unitIdx]
+ s.unitIdx++
+ if len(unit.windows) == 1 {
+ s.startWindow(&unit.windows[0])
+ continue
+ }
+ s.startMergeUnit(unit)
+ continue
+ }
+ seg := &s.cur.segments[s.segIdx]
+ s.segIdx++
+ if len(seg.cursors) > 1 {
+ s.stats.segmentsOverlap++
+ s.countSegment(false)
+ cursors := seg.cursors
+ seg.cursors = nil
+ s.merge = s.newMerge(cursors)
+ continue
+ }
+ bc, err := s.decodeSegment(ctx, seg)
+ if err != nil {
+ return nil, err
+ }
+ if bc == nil {
+ continue
+ }
+ s.stats.segmentsFast++
+ s.countSegment(true)
+ return s.lendCursor(bc), nil
+ }
+}
+
+// newMerge starts a merge over cursors, reusing the buffers of the last one.
+func (s *windowScan) newMerge(cursors []*blockCursor) *timeMerge {
+ if s.plan == nil {
+ s.plan = newColPlan(s.qr.batchSchema)
+ }
+ m := newTimeMerge(s.qr, &s.stats, s.loadCursor, s.plan, cursors, s.desc, s.spare)
+ m.pf = s.pf
+ s.spare = nil
+ return m
+}
+
+// finishMerge closes the running merge and keeps it as the spare for the next.
+func (s *windowScan) finishMerge() {
+ s.merge.close()
+ s.pf.setFrontier(0)
+ s.spare, s.merge = s.merge, nil
+}
+
+func (s *windowScan) startWindow(w *seriesWindow) {
+ s.cur = w
+ s.segIdx = 0
+ s.stats.windows++
+ overlapped := false
+ for i := range w.segments {
+ if len(w.segments[i].cursors) > 1 {
+ overlapped = true
+ break
+ }
+ }
+ if overlapped {
+ s.stats.windowsOverlapped++
+ }
+ if qm := s.qr.qm; qm != nil {
+ qm.scanWindows.Inc(1)
+ if overlapped {
+ qm.scanWindowsOverlapped.Inc(1)
+ }
+ }
+}
+
+// startMergeUnit starts the merge of windows whose rows interleave in time. The
+// windows are counted as overlapped: their blocks overlap by definition.
+func (s *windowScan) startMergeUnit(unit *scanUnit) {
+ var cursors []*blockCursor
+ for i := range unit.windows {
+ cursors = append(cursors, unit.windows[i].cursors...)
+ unit.windows[i].cursors = nil
+ }
+ n := int64(len(unit.windows))
+ s.stats.windows += n
+ s.stats.windowsOverlapped += n
+ s.stats.segmentsOverlap++
+ if qm := s.qr.qm; qm != nil {
+ qm.scanWindows.Inc(float64(n))
+ qm.scanWindowsOverlapped.Inc(float64(n))
+ }
+ s.countSegment(false)
+ s.cur = nil
+ s.merge = s.newMerge(cursors)
+}
+
+func (s *windowScan) countSegment(fast bool) {
+ qm := s.qr.qm
+ if qm == nil {
+ return
+ }
+ if fast {
+ qm.scanSegmentsFast.Inc(1)
+ return
+ }
+ qm.scanSegmentsOverlap.Inc(1)
+}
+
+// decodeSegment decodes the one cursor of seg and returns it, or nil when the block
+// has no row in the query time range. The scan owns the returned cursor; seg gives
+// it up.
+func (s *windowScan) decodeSegment(ctx context.Context, seg *scanSegment) (*blockCursor, error) {
+ bc := seg.cursors[0]
+ seg.cursors = nil
+ select {
+ case <-ctx.Done():
+ // A decode of bc may be running on a pool worker: withdraw or wait for it
+ // before the cursor is reset and pooled.
+ s.pf.discard(bc)
+ releaseBlockCursor(bc)
+ return nil, errors.WithMessagef(ctx.Err(), "interrupt: hit %d", s.qr.hit)
+ default:
+ }
+ ok, err := s.pf.take(ctx, bc)
+ if err != nil {
+ releaseBlockCursor(bc)
+ return nil, errors.WithMessagef(err, "interrupt: hit %d", s.qr.hit)
+ }
+ if !ok {
+ releaseBlockCursor(bc)
+ return nil, nil
+ }
+ return bc, nil
+}
+
+// lendCursor turns one decoded cursor into a batch and drops the scan's
+// reference; the batch's owners keep the cursor alive for as long as it lends
+// memory.
+func (s *windowScan) lendCursor(bc *blockCursor) *model.MeasureBatch {
+ qr := s.qr
+ var b *model.MeasureBatch
+ if len(bc.timestamps) > maxSelectionRows && s.desc {
+ // A descending block this large cannot be addressed by a uint16 selection.
+ b = newMeasureBatchForSchema(qr.batchSchema, len(bc.timestamps))
+ bc.copyAllToBatch(b, qr.batchSchema, qr.storedIndexValue, true)
+ } else {
+ b = bc.lendBatch(qr.batchSchema, qr.storedIndexValue, s.desc)
+ }
+ releaseBlockCursor(bc)
+ return b
+}
+
+// appendToBuilder copies every active row of piece onto the coalescing builder
+// and releases piece. The builder pins whatever owners piece pinned, since its
+// cells may still borrow from them. A row of another series than the row before it
+// starts a new series run.
+func (s *windowScan) appendToBuilder(piece *model.MeasureBatch) {
+ if s.builder == nil {
+ s.builder = newMeasureBatchForSchema(s.qr.batchSchema, coalesceTargetRows)
+ s.boundary = s.boundary[:0]
+ }
+ b := s.builder
+ b.Owners.PinAll(&piece.Owners)
+ copyRow := func(r int) {
+ if n := len(b.SeriesIDs); n > 0 && b.SeriesIDs[n-1] != piece.SeriesIDs[r] {
+ s.boundary = append(s.boundary, n)
+ }
+ b.Timestamps = append(b.Timestamps, piece.Timestamps[r])
+ b.Versions = append(b.Versions, piece.Versions[r])
+ b.ShardIDs = append(b.ShardIDs, piece.ShardIDs[r])
+ b.SeriesIDs = append(b.SeriesIDs, piece.SeriesIDs[r])
+ for i := range piece.Tags {
+ // Columns of one schema always share a type, so the copy cannot fail.
+ _ = vectorized.AppendColumnRange(b.Tags[i], piece.Tags[i], r, 1)
+ }
+ for i := range piece.Fields {
+ _ = vectorized.AppendColumnRange(b.Fields[i], piece.Fields[i], r, 1)
+ }
+ }
+ if piece.Selection != nil {
+ for _, r := range piece.Selection {
+ copyRow(int(r))
+ }
+ } else {
+ for r := range piece.RowCount() {
+ copyRow(r)
+ }
+ }
+ piece.Release()
+}
+
+// takeBuilder hands the coalesced batch to the caller.
+func (s *windowScan) takeBuilder() *model.MeasureBatch {
+ b := s.builder
+ s.builder = nil
+ if len(s.boundary) > 0 {
+ b.SeriesBoundaries = append(b.SeriesBoundaries[:0], s.boundary...)
+ b.SeriesBoundaries = append(b.SeriesBoundaries, b.RowCount())
+ } else {
+ // One series throughout: its entity tags are constant for the whole batch.
+ constifyEntityTags(b, s.qr.batchSchema)
+ }
+ s.boundary = s.boundary[:0]
+ s.stats.coalesced++
+ return b
+}
+
+// close releases every cursor and batch the scan still holds. It is idempotent.
+func (s *windowScan) close() {
+ // Stop the decodes first: no worker may read a cursor that is about to be released.
+ s.pf.close()
+ if s.merge != nil {
+ s.finishMerge()
+ }
+ s.spare = nil
+ for i := range s.units {
+ for j := range s.units[i].windows {
+ w := &s.units[i].windows[j]
+ releaseCursors(w.cursors)
+ w.cursors = nil
+ for k := range w.segments {
+ releaseCursors(w.segments[k].cursors)
+ w.segments[k].cursors = nil
+ }
+ }
+ }
+ s.units = nil
+ s.cur = nil
+ s.builder.Release()
+ s.builder = nil
+ s.ready.Release()
+ s.ready = nil
+}
+
+func releaseCursors(cursors []*blockCursor) {
+ for i, bc := range cursors {
+ if bc != nil {
+ releaseBlockCursor(bc)
+ cursors[i] = nil
+ }
+ }
+}
diff --git a/banyand/measure/scan_window_benchmark_test.go b/banyand/measure/scan_window_benchmark_test.go
new file mode 100644
index 000000000..08689c5e0
--- /dev/null
+++ b/banyand/measure/scan_window_benchmark_test.go
@@ -0,0 +1,85 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "testing"
+
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+)
+
+// BenchmarkVectorizedScan times only the storage scan: PullBatch until EOF over
+// the gate layouts, releasing every batch, with the window scan and with the
+// legacy decode-everything-then-merge scan in the same binary. Unlike
+// BenchmarkVectorizedStorage there is no pipeline or egress, whose per-row
+// allocations otherwise dominate and hide what the scan costs.
+//
+// go test ./banyand/measure -run '^$' -bench BenchmarkVectorizedScan -benchmem -count=6
+func BenchmarkVectorizedScan(b *testing.B) {
+ hs := newGateHarnesses(b)
+ scanOpts := model.MeasureQueryOptions{
+ TagProjection: gateProj(gateTSTag, gateTagEnv), FieldProjection: []string{gateFieldInt, gateFieldFlt},
+ }
+ layouts := []struct {
+ name string
+ layout gateLayout
+ timeOrdered bool
+ }{
+ {"single_series_10k", gateLayoutSingle, false},
+ {"multi_series_one_part", gateLayoutMulti, false},
+ {"multi_series_4parts_disjoint", gateLayoutDisjoint, false},
+ {"multi_series_2parts_overlap", gateLayoutOverlap, false},
+ // No ORDER BY: the series share one time range, so their rows interleave.
+ {"time_ordered_multi_series_one_part", gateLayoutMulti, true},
+ {"time_ordered_multi_series_4parts_disjoint", gateLayoutDisjoint, true},
+ {"time_ordered_multi_series_2parts_overlap", gateLayoutOverlap, true},
+ }
+ for _, l := range layouts {
+ h := hs.get(l.layout)
+ for _, legacy := range []bool{false, true} {
+ name := "window"
+ if legacy {
+ name = "legacy"
+ }
+ b.Run(l.name+"/"+name, func(b *testing.B) {
+ defer setLegacyBatchScan(setLegacyBatchScan(legacy))
+ schema := gateSchema(false, scanOpts)
+ b.ReportAllocs()
+ b.ResetTimer()
+ rows := 0
+ for b.Loop() {
+ qr := h.newQueryResult(schema, scanOpts, l.timeOrdered)
+ for {
+ mb, err := qr.PullBatch(context.Background())
+ if err != nil {
+ b.Fatal(err)
+ }
+ if mb == nil {
+ break
+ }
+ rows += mb.ActiveRows()
+ mb.Release()
+ }
+ qr.Release()
+ }
+ gateSink = rows
+ })
+ }
+ }
+}
diff --git a/banyand/measure/scan_window_fixture_test.go b/banyand/measure/scan_window_fixture_test.go
new file mode 100644
index 000000000..9145e43f2
--- /dev/null
+++ b/banyand/measure/scan_window_fixture_test.go
@@ -0,0 +1,379 @@
+// Licensed to Apache Software Foundation (ASF) under one or more contributor
+// license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright
+// ownership. Apache Software Foundation (ASF) licenses this file to you under
+// the Apache License, Version 2.0 (the "License"); you may
+// not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package measure
+
+import (
+ "context"
+ "fmt"
+ "slices"
+ "sync"
+ "testing"
+
+ "github.com/stretchr/testify/require"
+ "google.golang.org/protobuf/proto"
+
+ "github.com/apache/skywalking-banyandb/api/common"
+ databasev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/database/v1"
+ measurev1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/measure/v1"
+ modelv1 "github.com/apache/skywalking-banyandb/api/proto/banyandb/model/v1"
+ "github.com/apache/skywalking-banyandb/banyand/internal/storage"
+ "github.com/apache/skywalking-banyandb/pkg/convert"
+ pbv1 "github.com/apache/skywalking-banyandb/pkg/pb/v1"
+ "github.com/apache/skywalking-banyandb/pkg/query/model"
+ "github.com/apache/skywalking-banyandb/pkg/query/vectorized"
+ vmeasure "github.com/apache/skywalking-banyandb/pkg/query/vectorized/measure"
+)
+
+// Window-scan fixtures: memory parts built through the real memPart writer (so
+// blocks split, dedupe and encode exactly as flushed parts do) whose rows are a
+// closed-form function of (series, timestamp, version). A row's values change
+// with its version, so a query that kept the wrong version of a duplicated
+// timestamp reads visibly different tags and fields.
+
+const (
+ fxFamily = "default"
+ fxTagSvc = "svc" // string, few distinct values (dictionary encoded)
+ fxTagURL = "url" // string, unique per row (plain encoded)
+ fxTagN = "n" // int64 with nulls (bytes form)
+ fxTagN2 = "n2" // int64 without nulls (typed numeric form)
+ fxTagArr = "arr" // string array
+ fxTagIAr = "iarr" // int array
+ fxTagBin = "bin" // binary
+ fxTagHid = "hid" // string, substituted from the index in some tests
+ fxTagGho = "ghost" // not in the measure schema: passthrough column, always null
+ fxFldInt = "fi" // int64 field
+ fxFldFlt = "ff" // float64 field
+ fxFldNul = "fnull" // int64 field with nulls
+ fxFldStr = "fstr" // string field
+ fxFldBin = "fbin" // binary field
+)
+
+var fxTagTypes = map[string]pbv1.ValueType{
+ fxTagSvc: pbv1.ValueTypeStr, fxTagURL: pbv1.ValueTypeStr, fxTagN: pbv1.ValueTypeInt64, fxTagN2: pbv1.ValueTypeInt64,
+ fxTagArr: pbv1.ValueTypeStrArr, fxTagIAr: pbv1.ValueTypeInt64Arr, fxTagBin: pbv1.ValueTypeBinaryData, fxTagHid: pbv1.ValueTypeStr,
+}
+
+func fxMeasure() *databasev1.Measure {
+ return &databasev1.Measure{
+ // svc is a function of the series (svc-<sid mod 4>) and hid is the tag the
+ // tests substitute from the index per series (and is absent otherwise), so
+ // both are constant within a series: the entity tags a lent batch carries as
+ // ConstColumns.
+ Entity: &databasev1.Entity{TagNames: []string{fxTagSvc, fxTagHid}},
+ TagFamilies: []*databasev1.TagFamilySpec{{Name: fxFamily, Tags: []*databasev1.TagSpec{
+ {Name: fxTagSvc, Type: databasev1.TagType_TAG_TYPE_STRING},
+ {Name: fxTagURL, Type: databasev1.TagType_TAG_TYPE_STRING},
+ {Name: fxTagN, Type: databasev1.TagType_TAG_TYPE_INT},
+ {Name: fxTagN2, Type: databasev1.TagType_TAG_TYPE_INT},
+ {Name: fxTagArr, Type: databasev1.TagType_TAG_TYPE_STRING_ARRAY},
+ {Name: fxTagIAr, Type: databasev1.TagType_TAG_TYPE_INT_ARRAY},
+ {Name: fxTagBin, Type: databasev1.TagType_TAG_TYPE_DATA_BINARY},
+ {Name: fxTagHid, Type: databasev1.TagType_TAG_TYPE_STRING},
+ }}},
+ Fields: []*databasev1.FieldSpec{
+ {Name: fxFldInt, FieldType: databasev1.FieldType_FIELD_TYPE_INT},
+ {Name: fxFldFlt, FieldType: databasev1.FieldType_FIELD_TYPE_FLOAT},
+ {Name: fxFldNul, FieldType: databasev1.FieldType_FIELD_TYPE_INT},
+ {Name: fxFldStr, FieldType: databasev1.FieldType_FIELD_TYPE_STRING},
+ {Name: fxFldBin, FieldType: databasev1.FieldType_FIELD_TYPE_DATA_BINARY},
+ },
+ }
+}
+
+// fxRange is rows [from, to] of one series at one version.
+type fxRange struct {
+ sid common.SeriesID
+ from int64
+ to int64
+ version int64
+}
+
+// fxPart is one memory part: the ranges written into it.
+type fxPart []fxRange
+
+type fxFixture struct {
+ written []fxPart
+ parts []partRef
+ sids []common.SeriesID // sorted
+}
+
+func fxValue(name string, valueType pbv1.ValueType, value []byte) *nameValue {
+ return &nameValue{name: name, valueType: valueType, value: value}
+}
+
+func fxPoints(ranges fxPart) *dataPoints {
+ dps := &dataPoints{}
+ for _, r := range ranges {
+ for ts := r.from; ts <= r.to; ts++ {
+ sid, v := int64(r.sid), r.version
+ n := fxValue(fxTagN, pbv1.ValueTypeInt64, convert.Int64ToBytes(ts*100+v))
+ if ts%13 == 0 {
+ n.value = nil
+ }
+ arr := &nameValue{name: fxTagArr, valueType: pbv1.ValueTypeStrArr, valueArr: [][]byte{
+ []byte(fmt.Sprintf("a%d", ts%3)), []byte(fmt.Sprintf("has|pipe-%d", v)), []byte(`back\slash`),
+ }}
+ if ts%17 == 0 {
+ arr.valueArr = nil
+ arr.value = nil
+ }
+ iarr := &nameValue{name: fxTagIAr, valueType: pbv1.ValueTypeInt64Arr, valueArr: [][]byte{
+ convert.Int64ToBytes(ts), convert.Int64ToBytes(v),
+ }}
+ bin := fxValue(fxTagBin, pbv1.ValueTypeBinaryData, []byte(fmt.Sprintf("bin-%d-%d-%d", sid, ts, v)))
+ if ts%19 == 0 {
+ bin.value = nil
+ }
+ fnull := fxValue(fxFldNul, pbv1.ValueTypeInt64, convert.Int64ToBytes(ts-v))
+ if ts%7 == 0 {
+ fnull.value = nil
+ }
+ fstr := fxValue(fxFldStr, pbv1.ValueTypeStr, []byte(fmt.Sprintf("fs-%d-%d", ts, v)))
+ if ts%11 == 0 {
+ fstr.value = nil
+ }
+ dps.seriesIDs = append(dps.seriesIDs, r.sid)
+ dps.timestamps = append(dps.timestamps, ts)
+ dps.versions = append(dps.versions, v)
+ dps.tagFamilies = append(dps.tagFamilies, []nameValues{{name: fxFamily, values: []*nameValue{
+ fxValue(fxTagSvc, pbv1.ValueTypeStr, []byte(fmt.Sprintf("svc-%d", sid%4))),
+ fxValue(fxTagURL, pbv1.ValueTypeStr, []byte(fmt.Sprintf("/u/%d/%d/%d", sid, ts, v))),
+ n,
+ fxValue(fxTagN2, pbv1.ValueTypeInt64, convert.Int64ToBytes(sid*1000000+ts*10+v)),
+ arr,
+ iarr,
+ bin,
+ }}})
+ dps.fields = append(dps.fields, nameValues{name: "f", values: []*nameValue{
+ fxValue(fxFldInt, pbv1.ValueTypeInt64, convert.Int64ToBytes(sid*1000+ts*3+v)),
+ fxValue(fxFldFlt, pbv1.ValueTypeFloat64, convert.Float64ToBytes(float64(ts)*0.5+float64(v))),
+ fnull,
+ fstr,
+ fxValue(fxFldBin, pbv1.ValueTypeBinaryData, []byte(fmt.Sprintf("fb-%d-%d", ts, v))),
+ }})
+ }
+ }
+ return dps
+}
+
+// newFixture writes each part through memPart.mustInitFromDataPoints and opens it.
+// Parts are listed oldest first, the order a snapshot would hand them out in.
+func newFixture(t testing.TB, parts ...fxPart) *fxFixture {
+ t.Helper()
+ verifyNoScanLeaks(t)
+ fx := &fxFixture{written: parts}
+ seen := map[common.SeriesID]struct{}{}
+ for i, ranges := range parts {
+ mp := generateMemPart()
+ mp.mustInitFromDataPoints(fxPoints(ranges))
+ p := openMemPart(mp)
+ p.partMetadata.ID = uint64(i + 1)
+ fx.parts = append(fx.parts, partRef{p: p, cache: storage.NewShardCache("scan-window-test", 0, 0)})
+ for _, r := range ranges {
+ if _, ok := seen[r.sid]; !ok {
+ seen[r.sid] = struct{}{}
+ fx.sids = append(fx.sids, r.sid)
+ }
+ }
+ }
+ slices.Sort(fx.sids)
+ return fx
+}
+
+// fxQuery is one query over a fixture.
+type fxQuery struct {
+ groupBy *model.MeasureGroupBy
+ agg *model.MeasureAgg
+ stored map[common.SeriesID]map[string]*modelv1.TagValue
+ order []common.SeriesID // the order sids are requested in; nil means ascending
+ tags []string
+ fields []string
+ minTS int64
+ maxTS int64
+ offset uint32
+ limit uint32
+ orderByTS bool
+ ascTS bool
+ passthrough bool
+}
+
+func (q *fxQuery) options() model.MeasureQueryOptions {
+ var proj []model.TagProjection
+ if len(q.tags) > 0 {
+ proj = []model.TagProjection{{Family: fxFamily, Names: q.tags}}
+ }
+ return model.MeasureQueryOptions{TagProjection: proj, FieldProjection: q.fields, GroupBy: q.groupBy, Agg: q.agg}
+}
+
+func (q *fxQuery) schema(t testing.TB) *vectorized.BatchSchema {
+ t.Helper()
+ opts := q.options()
+ schema, err := vmeasure.BuildBatchSchema(fxMeasure(), opts)
+ require.NoError(t, err)
+ if q.passthrough {
+ return passthroughSchema(schema, opts)
+ }
+ return schema
+}
+
+var (
+ fixturePool *decodePool
+ fixturePoolOnce sync.Once
+)
+
+// sharedFixturePool is the decode pool every fixture query shares, as the queries of
+// a node share one, so the scan suites run with prefetch on. A test that depends on
+// the order decodes happen in sets qr.dp to nil before the scan is planned.
+func sharedFixturePool() *decodePool {
+ fixturePoolOnce.Do(func() { fixturePool = newDecodePool(4, nil) })
+ return fixturePool
+}
+
+// newResult builds a queryResult over the fixture the way searchBlocks does.
+func (fx *fxFixture) newResult(t testing.TB, q *fxQuery, schema *vectorized.BatchSchema) *queryResult {
+ t.Helper()
+ opts := q.options()
+ qo := queryOptions{schemaTagTypes: fxTagTypes, minTimestamp: q.minTS, maxTimestamp: q.maxTS}
+ qo.MeasureQueryOptions.TagProjection = opts.TagProjection
+ qo.MeasureQueryOptions.FieldProjection = opts.FieldProjection
+ order := q.order
+ if order == nil {
+ order = fx.sids
+ }
+ qr := &queryResult{
+ ctx: context.Background(),
+ dp: sharedFixturePool(),
+ batchSchema: schema,
+ tagProjection: opts.TagProjection,
+ storedIndexValue: q.stored,
+ orderByTS: q.orderByTS,
+ ascTS: q.ascTS,
+ sidToIndex: map[common.SeriesID]int{},
... 19918 lines suppressed ...