You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by fr...@apache.org on 2026/07/27 22:59:45 UTC
(kafka) branch trunk updated: KAFKA-20706: IQv2 TimestampedWindowRangeWithHeadersQuery (KIP-1356) (#22853)
This is an automated email from the ASF dual-hosted git repository.
frankvicky pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new 80014087110 KAFKA-20706: IQv2 TimestampedWindowRangeWithHeadersQuery (KIP-1356) (#22853)
80014087110 is described below
commit 8001408711059edc5dda803d2f50e551a7e7e7a9
Author: Jess668 <je...@confluent.io>
AuthorDate: Mon Jul 27 18:59:38 2026 -0400
KAFKA-20706: IQv2 TimestampedWindowRangeWithHeadersQuery (KIP-1356) (#22853)
## Summary
Adds the fourth and final IQv2 query type from
[KIP-1356](https://cwiki.apache.org/confluence/spaces/KAFKA/pages/430408653/KIP-1356+Introduce+IQv2+for+headers-aware+state+stores),
`TimestampedWindowRangeWithHeadersQuery` — the headers-aware parallel of
`WindowRangeQuery`. It returns a `ReadOnlyRecordIterator<Windowed<K>,
V>` whose elements carry the windowed key, value, event-time, and record
headers, and — like `WindowRangeQuery` — has two mutually exclusive
forms:
- `withWindowStartRange(timeFrom, timeTo)` is handled by window stores
and returns every key whose window start falls within the closed range
`[timeFrom, timeTo]`. Each element's `timestamp()` is the stored
record's event-time.
- `withKey(key)` is handled by session stores and returns every session
for that key. Session aggregations carry no per-record event-time, so
`timestamp()` is the session window's (inclusive) end.
Submitting the wrong form (`withKey` to a window store, or
`withWindowStartRange` to a session store) fails with
`UNKNOWN_QUERY_TYPE`, as does submitting the query to a store not built
with a `WithHeaders` builder.
## Implementation
Handling is done at the metered layer:
- `MeteredTimestampedWindowStoreWithHeaders` serves the
`withWindowStartRange` form by forwarding
a raw `WindowRangeQuery.withWindowStartRange` to the wrapped store and
surfacing each entry as a
`ReadOnlyRecord` (via `Record`) carrying the stored event-time and
headers, with the headers
frozen read-only. Because a `ReadOnlyRecord` timestamp is
contractually non-negative, an entry
with a negative stored timestamp (e.g. `NO_TIMESTAMP`/`-1` from a
`WithHeaders` store built over
a plain window supplier) cannot be represented, so advancing the
iterator throws
`StreamsException` at that entry. The `withKey` form is rejected.
- `MeteredSessionStoreWithHeaders` serves the `withKey` form by
forwarding a raw
`WindowRangeQuery.withKey` and surfacing each session as a
`ReadOnlyRecord` whose timestamp is
the session-window end. The `withWindowStartRange` form is rejected.
The native RocksDB header stores (`RocksDBSessionStoreWithHeaders`,
`RocksDBTimeOrderedSessionStoreWithHeaders`,
`RocksDBTimestampedWindowStoreWithHeaders`,
`RocksDBTimeOrderedWindowStoreWithHeaders`) previously overrode
`query()` to return `UNKNOWN_QUERY_TYPE` (the window stores
special-cased only `WindowKeyQuery`). Removing those overrides lets
them inherit native IQv2 handling via `StoreQueryUtils`, so the raw
`WindowRangeQuery` forwarded by the metered layer is served on native
builds. This also fixes the pre-existing gap where a plain
`WindowRangeQuery` was unsupported against a natively-built header
store.
Minor: `MeteredSessionStoreWithHeaders.query()` now records execution
info once after dispatch instead of per-branch, and both metered stores
seed the timing clock from `time.nanoseconds()` (matching the parent
stores) so execution-info durations are correct under a mock clock.
## Testing
Unit, store-builder, and end-to-end coverage across both forms and all
build paths (native, timestamped/plain adapter, in-memory):
- **Metered-store unit tests**
(`MeteredTimestampedWindowStoreWithHeadersTest`,
`MeteredSessionStoreWithHeadersTest`): bounds/key are forwarded as a
raw `WindowRangeQuery`; the
wrong form is rejected; wrapped-store failures propagate;
`num-open-iterators` is tracked,
including the intentionally non-idempotent double-close and leaving
the iterator open when
`next()` throws on a negative stored timestamp; returned values,
headers, and read-only
(immutable) headers are asserted.
- **RocksDB store tests**: the native stores now serve the forwarded raw
`WindowRangeQuery` /
`WindowKeyQuery` through their inherited `StoreQueryUtils` handling,
returning the raw stored
header-format bytes.
- **Store-builder tests** (`SessionStoreWithHeadersBuilderTest` (new),
`TimestampedWindowStoreWithHeadersBuilderTest`): builder wiring plus
end-to-end query behavior —
header round-trip and immutability, empty headers on the adapter
build, plain-supplier and
negative-timestamp throws, execution-info, and native-vs-adapter
parity with an interleaved
noise key and variable-length sessions.
- **Integration tests** (`IQv2HeadersStoreIntegrationTest`): end-to-end
IQv2 queries for both
forms through `KafkaStreams` — header round-trip, empty headers on the
adapter build,
position/caching semantics (a cache-only write is not visible under a
position bound), cross-form
and non-headers-store rejection, and variable-length merged session
windows.
Reviewers: Alieh Saeedi <as...@confluent.io>, TengYao Chi
<fr...@apache.org>
---
.../IQv2HeadersStoreIntegrationTest.java | 455 ++++++++++++++++++-
.../streams/processor/api/ReadOnlyRecord.java | 10 +-
.../TimestampedWindowRangeWithHeadersQuery.java | 158 +++++++
.../internals/MeteredSessionStoreWithHeaders.java | 137 +++++-
.../MeteredTimestampedWindowStoreWithHeaders.java | 136 ++++--
.../internals/RocksDBSessionStoreWithHeaders.java | 40 +-
.../RocksDBTimeOrderedSessionStoreWithHeaders.java | 40 +-
.../RocksDBTimeOrderedWindowStoreWithHeaders.java | 45 +-
.../RocksDBTimestampedWindowStoreWithHeaders.java | 45 +-
.../MeteredSessionStoreWithHeadersTest.java | 197 ++++++++
...teredTimestampedWindowStoreWithHeadersTest.java | 223 ++++++++-
.../RocksDBSessionStoreWithHeadersTest.java | 45 +-
...ksDBTimeOrderedSessionStoreWithHeadersTest.java | 45 +-
...cksDBTimeOrderedWindowStoreWithHeadersTest.java | 45 +-
...cksDBTimestampedWindowStoreWithHeadersTest.java | 69 +--
.../SessionStoreWithHeadersBuilderTest.java | 505 +++++++++++++++++++++
...mestampedWindowStoreWithHeadersBuilderTest.java | 254 ++++++++++-
17 files changed, 2201 insertions(+), 248 deletions(-)
diff --git a/streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/IQv2HeadersStoreIntegrationTest.java b/streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/IQv2HeadersStoreIntegrationTest.java
index a67e5784c88..4a3a12569cc 100644
--- a/streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/IQv2HeadersStoreIntegrationTest.java
+++ b/streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/IQv2HeadersStoreIntegrationTest.java
@@ -36,6 +36,7 @@ import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
import org.apache.kafka.streams.kstream.Consumed;
import org.apache.kafka.streams.kstream.Produced;
import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.SessionWindow;
import org.apache.kafka.streams.processor.api.Processor;
import org.apache.kafka.streams.processor.api.ProcessorContext;
import org.apache.kafka.streams.processor.api.ProcessorSupplier;
@@ -51,7 +52,12 @@ import org.apache.kafka.streams.query.StateQueryResult;
import org.apache.kafka.streams.query.TimestampedKeyWithHeadersQuery;
import org.apache.kafka.streams.query.TimestampedRangeWithHeadersQuery;
import org.apache.kafka.streams.query.TimestampedWindowKeyWithHeadersQuery;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
+import org.apache.kafka.streams.state.AggregationWithHeaders;
+import org.apache.kafka.streams.state.KeyValueIterator;
import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.SessionStoreWithHeaders;
import org.apache.kafka.streams.state.StoreBuilder;
import org.apache.kafka.streams.state.Stores;
import org.apache.kafka.streams.state.TimestampedKeyValueStore;
@@ -74,6 +80,7 @@ import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
+import java.util.Comparator;
import java.util.LinkedList;
import java.util.List;
import java.util.Properties;
@@ -93,8 +100,9 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
*
* <p>It builds a KIP-1271 {@code WithHeaders} store, writes records (with headers) into it through a processor,
* and queries it through IQv2. Covers {@link TimestampedKeyWithHeadersQuery},
- * {@link TimestampedRangeWithHeadersQuery}, and {@link TimestampedWindowKeyWithHeadersQuery}; the remaining
- * KIP-1356 headers query type (window range / session) is expected to extend this class as it lands.
+ * {@link TimestampedRangeWithHeadersQuery}, {@link TimestampedWindowKeyWithHeadersQuery}, and
+ * {@link TimestampedWindowRangeWithHeadersQuery} (both its window-store {@code withWindowStartRange}
+ * form and its session-store {@code withKey} form) -- all four KIP-1356 headers query types.
*/
@Tag("integration")
public class IQv2HeadersStoreIntegrationTest {
@@ -115,6 +123,13 @@ public class IQv2HeadersStoreIntegrationTest {
private static final Duration WINDOW_SIZE = Duration.ofMillis(WINDOW_SIZE_MS);
private static final Duration RETENTION = Duration.ofMinutes(1);
+ // Session-store test parameters (used by the session (window-range withKey) headers query tests).
+ // Each produced record becomes its own session spanning [timestamp, timestamp + SESSION_LENGTH_MS].
+ private static final long SESSION_LENGTH_MS = 10L;
+ // Inactivity gap used by the merging session processor: records for a key within this many ms of an
+ // existing session are merged into one (variable-length) session, mirroring session-window aggregation.
+ private static final long SESSION_MERGE_GAP_MS = 50L;
+
private String inputStream;
private String outputStream;
private long baseTimestamp;
@@ -539,6 +554,196 @@ public class IQv2HeadersStoreIntegrationTest {
assertWindowedRecord(records.get(0), 1, baseTimestamp, "one", new RecordHeaders());
}
+ @Test
+ public void shouldHandleTimestampedWindowRangeWithHeadersQuery() throws Exception {
+ startStreamsWithWindowHeadersStore();
+
+ final long window0 = baseTimestamp;
+ final long window1 = baseTimestamp + 100;
+ final long window2 = baseTimestamp + 200;
+
+ // key 1 across two windows: headers, then empty headers.
+ produceDataToTopicWithHeaders(inputStream, window0, HEADERS, KeyValue.pair(1, "v0"));
+ produceDataToTopicWithHeaders(inputStream, window1, new RecordHeaders(), KeyValue.pair(1, "v1"));
+ // Noise key 2: a live entry inside the queried range, a tombstone that must not affect key 1's
+ // data, and a live entry outside the queried range (window2), which must be excluded.
+ produceDataToTopicWithHeaders(inputStream, window0, HEADERS, KeyValue.pair(2, "o0"));
+ produceDataToTopicWithHeaders(inputStream, window1, HEADERS, KeyValue.pair(2, "o1"), KeyValue.pair(2, null));
+ produceDataToTopicWithHeaders(inputStream, window2, HEADERS, KeyValue.pair(2, "o2"));
+
+ // Range [window0, window1]: every key's windows in that start range.
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records =
+ windowRangeQuery(Instant.ofEpochMilli(window0), Instant.ofEpochMilli(window1));
+
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> key1Records = recordsForKey(records, 1);
+ assertEquals(List.of(window0, window1), windowStarts(key1Records));
+ assertWindowedRecord(key1Records.get(0), 1, window0, "v0", HEADERS);
+ assertWindowedRecord(key1Records.get(1), 1, window1, "v1", new RecordHeaders());
+
+ // key 2's window0 survives, window1 is tombstoned, window2 is excluded by the range bound.
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> key2Records = recordsForKey(records, 2);
+ assertEquals(List.of(window0), windowStarts(key2Records));
+ assertWindowedRecord(key2Records.get(0), 2, window0, "o0", HEADERS);
+ }
+
+ @Test
+ public void shouldNotSeeUnflushedWriteInWindowRangeQueryWhenCachingEnabled() throws Exception {
+ // Like the window key query, a window range query bypasses the record cache: CachingWindowStore
+ // does not intercept IQv2 queries, so the query is served straight from the persistent store.
+ // That store's Position only advances on writes that reach it, so a query bound on the input
+ // position never catches up while the write lives only in the cache.
+ commitIntervalMs = Duration.ofMinutes(10).toMillis();
+ startStreamsWithWindowHeadersStore(true);
+
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "v0"));
+
+ final StateQueryRequest<ReadOnlyRecordIterator<Windowed<Integer>, String>> request =
+ inStore(STORE_NAME)
+ .withQuery(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withWindowStartRange(
+ Instant.ofEpochMilli(baseTimestamp), Instant.ofEpochMilli(baseTimestamp)))
+ .withPositionBound(PositionBound.at(inputPosition));
+ final StateQueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> result = kafkaStreams.query(request);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> onlyResult = result.getOnlyPartitionResult();
+ assertTrue(onlyResult.isFailure(), "A window range query bound on the input position must not catch up while the write is cache-only");
+ assertEquals(FailureReason.NOT_UP_TO_BOUND, onlyResult.getFailureReason());
+ }
+
+ @Test
+ public void shouldFailWithUnknownQueryTypeForWindowRangeQueryAgainstNonHeadersStore() throws Exception {
+ startStreamsWithWindowNonHeadersStore();
+ assertUnknownQueryType(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withWindowStartRange(
+ Instant.ofEpochMilli(baseTimestamp), Instant.ofEpochMilli(baseTimestamp + 1)));
+ }
+
+ @Test
+ public void shouldThrowForTimestampedWindowRangeWithHeadersQueryOnPlainSupplier() throws Exception {
+ // A WithHeaders window builder over a plain (non-timestamped) window supplier: entries come back
+ // with timestamp = -1. The query succeeds, but iterating throws because -1 cannot be a ReadOnlyRecord.
+ startStreamsWithWindowPlainSupplierStore();
+
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "one"));
+
+ final StateQueryRequest<ReadOnlyRecordIterator<Windowed<Integer>, String>> request =
+ inStore(STORE_NAME)
+ .withQuery(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withWindowStartRange(
+ Instant.ofEpochMilli(baseTimestamp), Instant.ofEpochMilli(baseTimestamp + 1)))
+ .withPositionBound(PositionBound.at(inputPosition));
+ final StateQueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> result =
+ IntegrationTestUtils.iqv2WaitForResult(kafkaStreams, request);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> onlyResult = result.getOnlyPartitionResult();
+ assertTrue(onlyResult.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<Integer>, String> iterator = onlyResult.getResult()) {
+ final StreamsException e = assertThrows(StreamsException.class, iterator::next);
+ assertTrue(e.getMessage().contains("as a ReadOnlyRecord") && e.getMessage().contains("is negative"),
+ "unexpected message: " + e.getMessage());
+ }
+ }
+
+ @Test
+ public void shouldReturnEmptyHeadersForTimestampedWindowRangeWithHeadersQueryOnAdapterStore() throws Exception {
+ // A WithHeaders window builder over a plain *timestamped* window supplier keeps timestamps but
+ // drops headers via TimestampedToHeadersWindowStoreAdapter. A window range query reads the
+ // underlying store directly, so headers come back empty (never null), even though they were written.
+ startStreamsWithWindowAdapterStore();
+
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "one"));
+
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records =
+ windowRangeQuery(Instant.ofEpochMilli(baseTimestamp), Instant.ofEpochMilli(baseTimestamp));
+ assertEquals(1, records.size());
+ assertWindowedRecord(records.get(0), 1, baseTimestamp, "one", new RecordHeaders());
+ }
+
+ @Test
+ public void shouldFailWithUnknownQueryTypeWhenWithKeySubmittedToWindowStore() throws Exception {
+ startStreamsWithWindowHeadersStore();
+ assertUnknownQueryTypeWithMessageFragment(
+ TimestampedWindowRangeWithHeadersQuery.<Integer, String>withKey(1),
+ "WindowStores only supports TimestampedWindowRangeWithHeadersQuery.withWindowStartRange");
+ }
+
+ @Test
+ public void shouldHandleTimestampedWindowRangeWithHeadersQueryOnSessionStore() throws Exception {
+ // Session windows are variable-length and grow by merging (unlike fixed-size window-store windows).
+ // Feed several records for key 1 within the inactivity gap so they collapse into one session
+ // spanning [base, base+55], then a distant record that forms a second, zero-length session. The
+ // query must return both variable-length sessions with the merged value and the last writer's headers.
+ startStreamsWithSessionHeadersStoreMerging();
+
+ // key 1: three records within SESSION_MERGE_GAP_MS merge into one session [base, base+55] = "abc";
+ // the merged session's headers come from the last record folded in (base+55, HEADERS).
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "a"));
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp + 30, new RecordHeaders(), KeyValue.pair(1, "b"));
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp + 55, HEADERS, KeyValue.pair(1, "c"));
+ // A distant record (> gap away) starts a separate, zero-length session [base+500, base+500] = "d".
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp + 500, new RecordHeaders(), KeyValue.pair(1, "d"));
+ // Noise key 2: a session written then tombstoned -- must not leak into key 1's query nor survive its own.
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(2, "o"));
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, new RecordHeaders(), KeyValue.pair(2, null));
+
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records = sortedByWindowStart(sessionRangeQuery(1));
+ assertEquals(2, records.size());
+ // Merged, variable-length session (length 55, not a fixed size), value concatenated in arrival order.
+ assertSessionRecord(records.get(0), 1, baseTimestamp, baseTimestamp + 55, "abc", HEADERS);
+ // A separate zero-length session -- proves the end round-trips from the raw result, not a fixed size.
+ assertSessionRecord(records.get(1), 1, baseTimestamp + 500, baseTimestamp + 500, "d", new RecordHeaders());
+
+ // key 2's only session was tombstoned -> empty result; a never-written key is likewise empty.
+ assertTrue(sessionRangeQuery(2).isEmpty(), "expected key 2's tombstoned session to be excluded");
+ assertTrue(sessionRangeQuery(999).isEmpty(), "expected no records for a never-written key");
+ }
+
+ @Test
+ public void shouldNotSeeUnflushedWriteInWindowRangeQueryOnSessionStoreWhenCachingEnabled() throws Exception {
+ // A session range query bypasses the record cache the same way the window-store form does:
+ // CachingSessionStore does not intercept IQv2 queries. A query bound on the input position never
+ // catches up while the write lives only in the cache.
+ commitIntervalMs = Duration.ofMinutes(10).toMillis();
+ startStreamsWithSessionHeadersStore(true);
+
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "v0"));
+
+ final StateQueryRequest<ReadOnlyRecordIterator<Windowed<Integer>, String>> request =
+ inStore(STORE_NAME)
+ .withQuery(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withKey(1))
+ .withPositionBound(PositionBound.at(inputPosition));
+ final StateQueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> result = kafkaStreams.query(request);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> onlyResult = result.getOnlyPartitionResult();
+ assertTrue(onlyResult.isFailure(), "A session range query bound on the input position must not catch up while the write is cache-only");
+ assertEquals(FailureReason.NOT_UP_TO_BOUND, onlyResult.getFailureReason());
+ }
+
+ @Test
+ public void shouldFailWithUnknownQueryTypeForWindowRangeQueryAgainstNonHeadersSessionStore() throws Exception {
+ startStreamsWithSessionNonHeadersStore();
+ assertUnknownQueryType(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withKey(1));
+ }
+
+ @Test
+ public void shouldReturnEmptyHeadersForTimestampedWindowRangeWithHeadersQueryOnSessionAdapterStore() throws Exception {
+ // A WithHeaders session builder over a plain persistent session supplier keeps sessions but drops
+ // headers via SessionToHeadersStoreAdapter.
+ startStreamsWithSessionAdapterStore();
+
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "one"));
+
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records = sessionRangeQuery(1);
+ assertEquals(1, records.size());
+ assertSessionRecord(records.get(0), 1, baseTimestamp, "one", new RecordHeaders());
+ }
+
+ @Test
+ public void shouldFailWithUnknownQueryTypeWhenWithWindowStartRangeSubmittedToSessionStore() throws Exception {
+ startStreamsWithSessionHeadersStore();
+ assertUnknownQueryTypeWithMessageFragment(
+ TimestampedWindowRangeWithHeadersQuery.<Integer, String>withWindowStartRange(
+ Instant.ofEpochMilli(baseTimestamp), Instant.ofEpochMilli(baseTimestamp + 1)),
+ "SessionStores only support TimestampedWindowRangeWithHeadersQuery.withKey");
+ }
+
private void startStreams(final StoreBuilder<?> storeBuilder,
final ProcessorSupplier<Integer, String, Integer, String> processorSupplier) throws Exception {
final StreamsBuilder builder = new StreamsBuilder();
@@ -652,6 +857,55 @@ public class IQv2HeadersStoreIntegrationTest {
WindowHeadersStoreWriterProcessor::new);
}
+ private void startStreamsWithSessionHeadersStore() throws Exception {
+ // Caching disabled: every IQv2 query is forced down to the persistent
+ // RocksDBSessionStoreWithHeaders layer.
+ startStreamsWithSessionHeadersStore(false);
+ }
+
+ private void startStreamsWithSessionHeadersStore(final boolean cachingEnabled) throws Exception {
+ final StoreBuilder<SessionStoreWithHeaders<Integer, String>> storeBuilder =
+ Stores.sessionStoreWithHeadersBuilder(
+ Stores.persistentSessionStoreWithHeaders(STORE_NAME, RETENTION),
+ Serdes.Integer(),
+ Serdes.String());
+ startStreams(
+ cachingEnabled ? storeBuilder.withCachingEnabled() : storeBuilder.withCachingDisabled(),
+ SessionHeadersStoreWriterProcessor::new);
+ }
+
+ private void startStreamsWithSessionHeadersStoreMerging() throws Exception {
+ final StoreBuilder<SessionStoreWithHeaders<Integer, String>> storeBuilder =
+ Stores.sessionStoreWithHeadersBuilder(
+ Stores.persistentSessionStoreWithHeaders(STORE_NAME, RETENTION),
+ Serdes.Integer(),
+ Serdes.String());
+ startStreams(storeBuilder.withCachingDisabled(), SessionMergingHeadersStoreWriterProcessor::new);
+ }
+
+ private void startStreamsWithSessionNonHeadersStore() throws Exception {
+ // A plain (non-WithHeaders) session store: the headers-aware query types are unsupported here.
+ startStreams(
+ Stores.sessionStoreBuilder(
+ Stores.persistentSessionStore(STORE_NAME, RETENTION),
+ Serdes.Integer(),
+ Serdes.String()),
+ SessionPlainStoreWriterProcessor::new);
+ }
+
+ private void startStreamsWithSessionAdapterStore() throws Exception {
+ // A WithHeaders session builder over a plain persistent session supplier: sessions persist, but
+ // headers are dropped via SessionToHeadersStoreAdapter. Unlike window/key-value stores, session
+ // stores have no timestamped/non-timestamped distinction, so there is no plain-supplier
+ // (negative-timestamp) variant to cover here.
+ startStreams(
+ Stores.sessionStoreWithHeadersBuilder(
+ Stores.persistentSessionStore(STORE_NAME, RETENTION),
+ Serdes.Integer(),
+ Serdes.String()).withCachingDisabled(),
+ SessionHeadersStoreWriterProcessor::new);
+ }
+
private <R> void assertUnknownQueryTypeAgainstNonHeadersStore(final Query<R> query) throws Exception {
startStreamsWithKeyValueNonHeadersStore();
assertUnknownQueryType(query);
@@ -668,6 +922,23 @@ public class IQv2HeadersStoreIntegrationTest {
assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getOnlyPartitionResult().getFailureReason());
}
+ // Like assertUnknownQueryType, but also asserts the failure message contains messageFragment -- used
+ // for the cross-form rejection tests (e.g. TimestampedWindowRangeWithHeadersQuery.withKey submitted
+ // to a window store, or withWindowStartRange submitted to a session store).
+ private <R> void assertUnknownQueryTypeWithMessageFragment(final Query<R> query, final String messageFragment) {
+ produceDataToTopicWithHeaders(inputStream, baseTimestamp, HEADERS, KeyValue.pair(1, "a0"));
+
+ final StateQueryRequest<R> request =
+ inStore(STORE_NAME).withQuery(query).withPositionBound(PositionBound.at(inputPosition));
+ final StateQueryResult<R> result = IntegrationTestUtils.iqv2WaitForResult(kafkaStreams, request);
+
+ final QueryResult<R> onlyResult = result.getOnlyPartitionResult();
+ assertTrue(onlyResult.isFailure());
+ assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, onlyResult.getFailureReason());
+ assertTrue(onlyResult.getFailureMessage().contains(messageFragment),
+ "unexpected message: " + onlyResult.getFailureMessage());
+ }
+
private ReadOnlyRecord<Integer, String> keyQuery(final int key) {
final StateQueryRequest<ReadOnlyRecord<Integer, String>> request =
inStore(STORE_NAME)
@@ -728,6 +999,60 @@ public class IQv2HeadersStoreIntegrationTest {
return records;
}
+ private List<ReadOnlyRecord<Windowed<Integer>, String>> windowRangeQuery(final Instant timeFrom,
+ final Instant timeTo) {
+ final StateQueryRequest<ReadOnlyRecordIterator<Windowed<Integer>, String>> request =
+ inStore(STORE_NAME)
+ .withQuery(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withWindowStartRange(timeFrom, timeTo))
+ .withPositionBound(PositionBound.at(inputPosition));
+ final StateQueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> result =
+ IntegrationTestUtils.iqv2WaitForResult(kafkaStreams, request);
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records = new ArrayList<>();
+ try (ReadOnlyRecordIterator<Windowed<Integer>, String> iterator = result.getOnlyPartitionResult().getResult()) {
+ while (iterator.hasNext()) {
+ records.add(iterator.next());
+ }
+ }
+ return records;
+ }
+
+ private List<ReadOnlyRecord<Windowed<Integer>, String>> sessionRangeQuery(final int key) {
+ final StateQueryRequest<ReadOnlyRecordIterator<Windowed<Integer>, String>> request =
+ inStore(STORE_NAME)
+ .withQuery(TimestampedWindowRangeWithHeadersQuery.<Integer, String>withKey(key))
+ .withPositionBound(PositionBound.at(inputPosition));
+ final StateQueryResult<ReadOnlyRecordIterator<Windowed<Integer>, String>> result =
+ IntegrationTestUtils.iqv2WaitForResult(kafkaStreams, request);
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records = new ArrayList<>();
+ try (ReadOnlyRecordIterator<Windowed<Integer>, String> iterator = result.getOnlyPartitionResult().getResult()) {
+ while (iterator.hasNext()) {
+ records.add(iterator.next());
+ }
+ }
+ return records;
+ }
+
+ // The raw window-range result is ordered key-major (by serialized key bytes) then by window start
+ // within each key, not globally by window start across keys -- so tests that mix multiple keys
+ // filter down to one key's sub-list before asserting window-start order.
+ private static List<ReadOnlyRecord<Windowed<Integer>, String>> recordsForKey(
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records, final int key) {
+ return records.stream().filter(r -> r.key().key() == key).collect(Collectors.toList());
+ }
+
+ private static List<Long> windowStarts(final List<ReadOnlyRecord<Windowed<Integer>, String>> records) {
+ return records.stream().map(r -> r.key().window().start()).collect(Collectors.toList());
+ }
+
+ // Session range results aren't guaranteed to come back in window-start order, so sort explicitly
+ // before asserting on them.
+ private static List<ReadOnlyRecord<Windowed<Integer>, String>> sortedByWindowStart(
+ final List<ReadOnlyRecord<Windowed<Integer>, String>> records) {
+ return records.stream()
+ .sorted(Comparator.comparingLong(r -> r.key().window().start()))
+ .collect(Collectors.toList());
+ }
+
private void assertWindowedRecord(final ReadOnlyRecord<Windowed<Integer>, String> record,
final int key,
final long windowStart,
@@ -745,6 +1070,34 @@ public class IQv2HeadersStoreIntegrationTest {
assertThrows(IllegalStateException.class, () -> record.headers().remove("source"));
}
+ private void assertSessionRecord(final ReadOnlyRecord<Windowed<Integer>, String> record,
+ final int key,
+ final long sessionStart,
+ final String value,
+ final Headers expectedHeaders) {
+ // Fixed-length sessions (one record == one session of SESSION_LENGTH_MS).
+ assertSessionRecord(record, key, sessionStart, sessionStart + SESSION_LENGTH_MS, value, expectedHeaders);
+ }
+
+ private void assertSessionRecord(final ReadOnlyRecord<Windowed<Integer>, String> record,
+ final int key,
+ final long sessionStart,
+ final long sessionEnd,
+ final String value,
+ final Headers expectedHeaders) {
+ assertEquals(Integer.valueOf(key), record.key().key());
+ assertEquals(sessionStart, record.key().window().start());
+ assertEquals(sessionEnd, record.key().window().end());
+ // timestamp() is sourced from the session window's end (AggregationWithHeaders carries no
+ // timestamp of its own), unlike the window-store form's stored event-time.
+ assertEquals(sessionEnd, record.timestamp());
+ assertEquals(value, record.value());
+ assertEquals(expectedHeaders, record.headers());
+ // The IQ result is a read-only snapshot: neither add nor remove is allowed.
+ assertThrows(IllegalStateException.class, () -> record.headers().add("x", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("source"));
+ }
+
private static List<Integer> keys(final List<ReadOnlyRecord<Integer, String>> records) {
return records.stream().map(ReadOnlyRecord::key).collect(Collectors.toList());
}
@@ -880,4 +1233,102 @@ public class IQv2HeadersStoreIntegrationTest {
context.forward(record);
}
}
+
+ private static class SessionHeadersStoreWriterProcessor implements Processor<Integer, String, Integer, String> {
+ private ProcessorContext<Integer, String> context;
+ private SessionStoreWithHeaders<Integer, String> store;
+
+ @Override
+ public void init(final ProcessorContext<Integer, String> context) {
+ this.context = context;
+ store = context.getStateStore(STORE_NAME);
+ }
+
+ @Override
+ public void process(final Record<Integer, String> record) {
+ // Each record becomes its own session, spanning [timestamp, timestamp + SESSION_LENGTH_MS].
+ // A null value tombstones that session -- via put(..., null), not remove(): unlike put(),
+ // remove() does not advance the store's IQv2 Position (AbstractRocksDBSegmentedBytesStore
+ // only calls StoreQueryUtils.updatePosition from put()), so a query bound on the input
+ // position would never catch up past a tombstone written via remove().
+ final Windowed<Integer> sessionKey =
+ new Windowed<>(record.key(), new SessionWindow(record.timestamp(), record.timestamp() + SESSION_LENGTH_MS));
+ store.put(sessionKey, record.value() == null ? null : AggregationWithHeaders.make(record.value(), record.headers()));
+ context.forward(record);
+ }
+ }
+
+ // Emulates session-window aggregation: each record is merged into any existing session for its key
+ // within SESSION_MERGE_GAP_MS on either side, producing a single variable-length session spanning
+ // [min(starts), max(ends)] whose value is the concatenation of the merged values. This is how real
+ // session aggregation (KStreamSessionWindowAggregate) grows and merges sessions, so it exercises the
+ // headers query against genuinely variable-length, merged sessions rather than fixed-size ones.
+ private static class SessionMergingHeadersStoreWriterProcessor implements Processor<Integer, String, Integer, String> {
+ private ProcessorContext<Integer, String> context;
+ private SessionStoreWithHeaders<Integer, String> store;
+
+ @Override
+ public void init(final ProcessorContext<Integer, String> context) {
+ this.context = context;
+ store = context.getStateStore(STORE_NAME);
+ }
+
+ @Override
+ public void process(final Record<Integer, String> record) {
+ final long ts = record.timestamp();
+ final List<KeyValue<Windowed<Integer>, AggregationWithHeaders<String>>> overlapping = new ArrayList<>();
+ try (final KeyValueIterator<Windowed<Integer>, AggregationWithHeaders<String>> sessions =
+ store.findSessions(record.key(), ts - SESSION_MERGE_GAP_MS, ts + SESSION_MERGE_GAP_MS)) {
+ while (sessions.hasNext()) {
+ overlapping.add(sessions.next());
+ }
+ }
+
+ if (record.value() == null) {
+ // Tombstone every overlapping session. Use put(..., null) (not remove()) so the store's
+ // IQv2 Position still advances -- remove() does not update Position.
+ for (final KeyValue<Windowed<Integer>, AggregationWithHeaders<String>> existing : overlapping) {
+ store.put(existing.key, null);
+ }
+ context.forward(record);
+ return;
+ }
+
+ long mergedStart = ts;
+ long mergedEnd = ts;
+ final StringBuilder mergedValue = new StringBuilder();
+ for (final KeyValue<Windowed<Integer>, AggregationWithHeaders<String>> existing : overlapping) {
+ mergedStart = Math.min(mergedStart, existing.key.window().start());
+ mergedEnd = Math.max(mergedEnd, existing.key.window().end());
+ mergedValue.append(existing.value.aggregation());
+ store.remove(existing.key); // superseded by the merged window written below
+ }
+ mergedValue.append(record.value());
+ store.put(
+ new Windowed<>(record.key(), new SessionWindow(mergedStart, mergedEnd)),
+ AggregationWithHeaders.make(mergedValue.toString(), record.headers()));
+ context.forward(record);
+ }
+ }
+
+ private static class SessionPlainStoreWriterProcessor implements Processor<Integer, String, Integer, String> {
+ private ProcessorContext<Integer, String> context;
+ private SessionStore<Integer, String> store;
+
+ @Override
+ public void init(final ProcessorContext<Integer, String> context) {
+ this.context = context;
+ store = context.getStateStore(STORE_NAME);
+ }
+
+ @Override
+ public void process(final Record<Integer, String> record) {
+ // See SessionHeadersStoreWriterProcessor: tombstone via put(..., null), not remove(), so the
+ // store's IQv2 Position advances.
+ final Windowed<Integer> sessionKey =
+ new Windowed<>(record.key(), new SessionWindow(record.timestamp(), record.timestamp() + SESSION_LENGTH_MS));
+ store.put(sessionKey, record.value());
+ context.forward(record);
+ }
+ }
}
diff --git a/streams/src/main/java/org/apache/kafka/streams/processor/api/ReadOnlyRecord.java b/streams/src/main/java/org/apache/kafka/streams/processor/api/ReadOnlyRecord.java
index c5162e46fd2..e5810808c64 100644
--- a/streams/src/main/java/org/apache/kafka/streams/processor/api/ReadOnlyRecord.java
+++ b/streams/src/main/java/org/apache/kafka/streams/processor/api/ReadOnlyRecord.java
@@ -54,11 +54,11 @@ public interface ReadOnlyRecord<K, V> {
/**
* The headers of the record. Never null.
*
- * <p>The returned {@link Headers} is part of a read-only view and must not be mutated by
- * callers.
+ * <p>The returned {@link Headers} should be treated as read-only. Records served as IQv2 results
+ * from a state store have their headers frozen via {@code RecordHeaders.setReadOnly()}, so adding
+ * or removing a header throws {@link IllegalStateException}. Freezing is not a deep copy, however:
+ * the {@code value()} bytes of an existing header can still be mutated in place, so callers must
+ * not modify them.
*/
- // TODO (KIP-1356 follow-up): once the IQv2 query types land, records served as IQv2
- // results from a state store will have their headers frozen via RecordHeaders.setReadOnly(),
- // so that any attempt to mutate them throws IllegalStateException.
Headers headers();
}
diff --git a/streams/src/main/java/org/apache/kafka/streams/query/TimestampedWindowRangeWithHeadersQuery.java b/streams/src/main/java/org/apache/kafka/streams/query/TimestampedWindowRangeWithHeadersQuery.java
new file mode 100644
index 00000000000..3e0ff40a2a2
--- /dev/null
+++ b/streams/src/main/java/org/apache/kafka/streams/query/TimestampedWindowRangeWithHeadersQuery.java
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.query;
+
+import org.apache.kafka.common.annotation.InterfaceAudience;
+import org.apache.kafka.common.annotation.InterfaceStability.Evolving;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.api.ReadOnlyRecord;
+import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
+import org.apache.kafka.streams.state.SessionStoreWithHeaders;
+import org.apache.kafka.streams.state.TimestampedWindowStoreWithHeaders;
+
+import java.time.Instant;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Interactive query for retrieving records across a window-start range or for a single key's
+ * sessions, including their record headers, from a {@link TimestampedWindowStoreWithHeaders} or a
+ * {@link SessionStoreWithHeaders}.
+ *
+ * <p>This is the headers-aware parallel of {@link WindowRangeQuery}: it returns a
+ * {@link ReadOnlyRecordIterator} of {@link ReadOnlyRecord} elements, each carrying the windowed key,
+ * value, timestamp, and headers, whereas {@link WindowRangeQuery} returns a plain
+ * {@link org.apache.kafka.streams.state.KeyValueIterator} of values (no headers). Like
+ * {@link WindowRangeQuery}, this query has two mutually exclusive forms:
+ *
+ * <ul>
+ * <li>{@link #withWindowStartRange(Instant, Instant)} is handled by window stores: it retrieves
+ * every key whose window start falls within the closed range {@code [timeFrom, timeTo]}. Each
+ * element's key is a {@link Windowed} describing the record's window, and
+ * {@link ReadOnlyRecord#timestamp()} is the stored record's event-time. A stored event-time is
+ * contractually non-negative; if the backing store does not persist timestamps -- for example a
+ * {@code WithHeaders} store built over a plain window-store supplier that surfaces entries with
+ * {@code NO_TIMESTAMP} ({@code -1}) -- that entry cannot be represented, so advancing the
+ * returned {@link ReadOnlyRecordIterator} throws
+ * {@link org.apache.kafka.streams.errors.StreamsException} at that entry. Because iteration can
+ * therefore throw mid-stream, always close the returned iterator (for example with
+ * try-with-resources), even when a call to {@code next()} throws; otherwise the underlying store
+ * iterator leaks and the {@code num-open-iterators} metric stays incremented.</li>
+ *
+ * <li>{@link #withKey(Object)} is handled by session stores: it retrieves every session for the
+ * given key. Each element's key is a {@link Windowed} whose window is the session's window.
+ * Session aggregations carry no per-record event-time of their own, so
+ * {@link ReadOnlyRecord#timestamp()} is filled from the session window's (inclusive) end
+ * timestamp. That value is validated non-negative when the window is constructed, so -- unlike
+ * the window-store form above -- this form can never throw while iterating: a session whose
+ * stored value deserializes to null is still returned, with {@link ReadOnlyRecord#value()}
+ * {@code null} (the window-store form instead throws for such an entry, as it has no
+ * event-time to surface).</li>
+ * </ul>
+ *
+ * <p>Submitting the {@code withWindowStartRange} form to a session store, or the {@code withKey} form
+ * to a window store, fails with {@link FailureReason#UNKNOWN_QUERY_TYPE}.
+ *
+ * <p>Headers are persisted and returned only when the store is backed by a native headers store,
+ * i.e. built with a KIP-1271 {@code WithHeaders} byte-store supplier (e.g.
+ * {@code Stores.persistentTimestampedWindowStoreWithHeaders} or
+ * {@code Stores.persistentSessionStoreWithHeaders}). A {@code WithHeaders} store built over a legacy
+ * (non-headers) supplier cannot persist headers, so the store-served reads come back with empty
+ * {@code headers()}.
+ *
+ * <p>Against a plain store not built with the {@code WithHeaders} builder at all, this query type is
+ * unsupported and fails with {@link FailureReason#UNKNOWN_QUERY_TYPE}.
+ *
+ * @param <K> Type of keys
+ * @param <V> Type of values
+ */
+@Evolving
+@InterfaceAudience.Public
+public final class TimestampedWindowRangeWithHeadersQuery<K, V> implements Query<ReadOnlyRecordIterator<Windowed<K>, V>> {
+
+ private final Optional<K> key;
+ private final Optional<Instant> timeFrom;
+ private final Optional<Instant> timeTo;
+
+ private TimestampedWindowRangeWithHeadersQuery(final Optional<K> key,
+ final Optional<Instant> timeFrom,
+ final Optional<Instant> timeTo) {
+ this.key = key;
+ this.timeFrom = timeFrom;
+ this.timeTo = timeTo;
+ }
+
+ /**
+ * Creates a query that will retrieve the records (value, timestamp, and headers) of every key
+ * whose window start falls within the closed range {@code [timeFrom, timeTo]}. Handled by window
+ * stores.
+ * @param timeFrom The inclusive lower bound of the window-start range
+ * @param timeTo The inclusive upper bound of the window-start range
+ * @param <K> The type of the key
+ * @param <V> The type of the value that will be retrieved
+ */
+ public static <K, V> TimestampedWindowRangeWithHeadersQuery<K, V> withWindowStartRange(final Instant timeFrom,
+ final Instant timeTo) {
+ Objects.requireNonNull(timeFrom, "timeFrom should not be null");
+ Objects.requireNonNull(timeTo, "timeTo should not be null");
+ return new TimestampedWindowRangeWithHeadersQuery<>(Optional.empty(), Optional.of(timeFrom), Optional.of(timeTo));
+ }
+
+ /**
+ * Creates a query that will retrieve the sessions (aggregation, timestamp, and headers) for the
+ * given {@code key}. Handled by session stores.
+ * @param key The key to retrieve
+ * @param <K> The type of the key
+ * @param <V> The type of the value that will be retrieved
+ */
+ public static <K, V> TimestampedWindowRangeWithHeadersQuery<K, V> withKey(final K key) {
+ Objects.requireNonNull(key, "the key should not be null");
+ return new TimestampedWindowRangeWithHeadersQuery<>(Optional.of(key), Optional.empty(), Optional.empty());
+ }
+
+ /**
+ * The key that was specified for this query, if this is a {@link #withKey(Object)} query.
+ */
+ public Optional<K> key() {
+ return key;
+ }
+
+ /**
+ * The inclusive lower bound of the window-start range, if this is a
+ * {@link #withWindowStartRange(Instant, Instant)} query.
+ */
+ public Optional<Instant> timeFrom() {
+ return timeFrom;
+ }
+
+ /**
+ * The inclusive upper bound of the window-start range, if this is a
+ * {@link #withWindowStartRange(Instant, Instant)} query.
+ */
+ public Optional<Instant> timeTo() {
+ return timeTo;
+ }
+
+ @Override
+ public String toString() {
+ return "TimestampedWindowRangeWithHeadersQuery{" +
+ "key=" + key +
+ ", timeFrom=" + timeFrom +
+ ", timeTo=" + timeTo +
+ '}';
+ }
+}
diff --git a/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeaders.java b/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeaders.java
index 65758c49591..e07051ad1ca 100644
--- a/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeaders.java
+++ b/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeaders.java
@@ -25,6 +25,8 @@ import org.apache.kafka.common.utils.Time;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.errors.ProcessorStateException;
import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.api.ReadOnlyRecord;
+import org.apache.kafka.streams.processor.api.Record;
import org.apache.kafka.streams.processor.internals.ProcessorRecordContext;
import org.apache.kafka.streams.processor.internals.SerdeGetter;
import org.apache.kafka.streams.query.FailureReason;
@@ -32,10 +34,12 @@ import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.Query;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
import org.apache.kafka.streams.query.WindowRangeQuery;
import org.apache.kafka.streams.query.internals.InternalQueryResultUtil;
import org.apache.kafka.streams.state.AggregationWithHeaders;
import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
import org.apache.kafka.streams.state.ReadOnlySessionStore;
import org.apache.kafka.streams.state.SessionStore;
import org.apache.kafka.streams.state.SessionStoreWithHeaders;
@@ -183,21 +187,28 @@ public class MeteredSessionStoreWithHeaders<K, AGG>
final PositionBound positionBound,
final QueryConfig config
) {
- final long start = config.isCollectExecutionInfo() ? System.nanoTime() : -1L;
+ final long start = time.nanoseconds();
final QueryResult<R> result;
+ // Queries this store handles itself go through its serdes; anything delegated to the wrapped
+ // store does not, so the two cases report execution info differently.
+ final boolean handledLocally;
if (query instanceof WindowRangeQuery) {
result = runRangeQuery((WindowRangeQuery<K, AGG>) query, positionBound, config);
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + getClass() + " with serdes " + serdes + " in " + (time.nanoseconds() - start) + "ns");
- }
+ handledLocally = true;
+ } else if (query instanceof TimestampedWindowRangeWithHeadersQuery) {
+ result = runTimestampedWindowRangeWithHeadersQuery((TimestampedWindowRangeWithHeadersQuery<K, AGG>) query, positionBound, config);
+ handledLocally = true;
} else {
result = wrapped().query(query, positionBound, config);
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + getClass() + " in " + (time.nanoseconds() - start) + "ns");
- }
+ handledLocally = false;
+ }
+
+ if (config.isCollectExecutionInfo()) {
+ result.addExecutionInfo(
+ "Handled in " + getClass()
+ + (handledLocally ? " with serdes " + serdes : "")
+ + " in " + (time.nanoseconds() - start) + "ns");
}
return result;
}
@@ -362,6 +373,47 @@ public class MeteredSessionStoreWithHeaders<K, AGG>
return queryResult;
}
+ /**
+ * Handles the {@code withKey} form of {@link TimestampedWindowRangeWithHeadersQuery} by
+ * forwarding a raw byte-level {@link WindowRangeQuery#withKey(Object)} to the wrapped store and
+ * surfacing each session as a {@link ReadOnlyRecord} (via {@link Record}) whose key is a
+ * {@link Windowed} of the deserialized key and the session's window, carrying the aggregation,
+ * headers, and a timestamp sourced from the session window's end. The {@code withWindowStartRange}
+ * form is rejected: it is handled by window stores, not session stores.
+ */
+ @SuppressWarnings("unchecked")
+ private <R> QueryResult<R> runTimestampedWindowRangeWithHeadersQuery(
+ final TimestampedWindowRangeWithHeadersQuery<K, AGG> query,
+ final PositionBound positionBound,
+ final QueryConfig config
+ ) {
+ final QueryResult<R> queryResult;
+
+ if (query.key().isPresent()) {
+ final WindowRangeQuery<Bytes, byte[]> rawKeyQuery =
+ WindowRangeQuery.withKey(serializeKey(query.key().get(), internalContext.headers()));
+ final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> rawResult =
+ wrapped().query(rawKeyQuery, positionBound, config);
+ if (rawResult.isSuccess()) {
+ final ReadOnlyRecordIterator<Windowed<K>, AGG> resultIterator =
+ new MeteredSessionWithHeadersReadOnlyRecordIterator(rawResult.getResult());
+ queryResult = (QueryResult<R>) InternalQueryResultUtil.copyAndSubstituteDeserializedResult(rawResult, resultIterator);
+ } else {
+ queryResult = (QueryResult<R>) rawResult;
+ }
+ } else {
+ queryResult = QueryResult.forFailure(
+ FailureReason.UNKNOWN_QUERY_TYPE,
+ "This store (" + getClass() + ") doesn't know how to"
+ + " execute the given query (" + query + ") because"
+ + " SessionStores only support TimestampedWindowRangeWithHeadersQuery.withKey."
+ + " Contact the store maintainer if you need support"
+ + " for a new query type."
+ );
+ }
+ return queryResult;
+ }
+
@Override
public ReadOnlySessionStore<K, AggregationWithHeaders<AGG>> readOnly(final IsolationLevel isolationLevel) {
Objects.requireNonNull(isolationLevel, "isolationLevel cannot be null");
@@ -521,4 +573,71 @@ public class MeteredSessionStoreWithHeaders<K, AGG>
return cachedNext.key;
}
}
+
+ /**
+ * Iterator backing the {@code withKey} form of {@link TimestampedWindowRangeWithHeadersQuery}:
+ * yields each session as a {@link ReadOnlyRecord} (implemented by {@link Record}) whose key is a
+ * {@link Windowed} of the deserialized key and the session's window, carrying the aggregation and
+ * stored headers, with the headers frozen so a caller cannot mutate the read-only result.
+ *
+ * <p>Unlike the window-store range query's iterator, there is no negative-timestamp check here:
+ * {@link ReadOnlyRecord#timestamp()} is sourced from the session window's end, which is validated
+ * non-negative when the window is constructed, so this iterator's {@code next()} can never throw.
+ */
+ private class MeteredSessionWithHeadersReadOnlyRecordIterator
+ implements ReadOnlyRecordIterator<Windowed<K>, AGG>, MeteredIterator {
+
+ private final KeyValueIterator<Windowed<Bytes>, byte[]> iter;
+ private final long startNs;
+ private final long startTimestampMs;
+
+ private MeteredSessionWithHeadersReadOnlyRecordIterator(
+ final KeyValueIterator<Windowed<Bytes>, byte[]> iter
+ ) {
+ this.iter = iter;
+ this.startNs = time.nanoseconds();
+ this.startTimestampMs = time.milliseconds();
+ numOpenIterators.increment();
+ openIterators.add(this);
+ }
+
+ @Override
+ public long startTimestamp() {
+ return startTimestampMs;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return iter.hasNext();
+ }
+
+ @Override
+ public ReadOnlyRecord<Windowed<K>, AGG> next() {
+ final KeyValue<Windowed<Bytes>, byte[]> next = iter.next();
+ final AggregationWithHeaders<AGG> aggregationWithHeaders = deserializeValue(next.value);
+ final Headers headers = aggregationWithHeaders != null ? aggregationWithHeaders.headers() : new RecordHeaders();
+ final K key = deserializeKey(next.key.key().get(), headers);
+ final Windowed<K> windowedKey = new Windowed<>(key, next.key.window());
+ final Record<Windowed<K>, AGG> record = new Record<>(
+ windowedKey,
+ aggregationWithHeaders != null ? aggregationWithHeaders.aggregation() : null,
+ windowedKey.window().end(),
+ headers);
+ ((RecordHeaders) record.headers()).setReadOnly();
+ return record;
+ }
+
+ @Override
+ public void close() {
+ try {
+ iter.close();
+ } finally {
+ final long duration = time.nanoseconds() - startNs;
+ fetchSensor.record(duration);
+ iteratorDurationSensor.record(duration);
+ numOpenIterators.decrement();
+ openIterators.remove(this);
+ }
+ }
+ }
}
diff --git a/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeaders.java b/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeaders.java
index bb84cb542a2..7fc57916052 100644
--- a/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeaders.java
+++ b/streams/src/main/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeaders.java
@@ -38,6 +38,7 @@ import org.apache.kafka.streams.query.Query;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
import org.apache.kafka.streams.query.TimestampedWindowKeyWithHeadersQuery;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
import org.apache.kafka.streams.query.WindowKeyQuery;
import org.apache.kafka.streams.query.WindowRangeQuery;
import org.apache.kafka.streams.query.internals.InternalQueryResultUtil;
@@ -53,6 +54,7 @@ import org.apache.kafka.streams.state.WindowStoreIterator;
import java.time.Instant;
import java.util.Objects;
+import java.util.function.BiFunction;
import java.util.function.Function;
import static org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency;
@@ -143,26 +145,34 @@ public class MeteredTimestampedWindowStoreWithHeaders<K, V>
final PositionBound positionBound,
final QueryConfig config
) {
- final long start = config.isCollectExecutionInfo() ? System.nanoTime() : -1L;
+ final long start = time.nanoseconds();
final QueryResult<R> result;
+ // Queries this store handles itself go through its serdes; anything delegated to the wrapped
+ // store does not, so the two cases report execution info differently.
+ final boolean handledLocally;
if (query instanceof WindowKeyQuery) {
result = runWindowKeyQuery((WindowKeyQuery<K, ValueTimestampHeaders<V>>) query, positionBound, config);
+ handledLocally = true;
} else if (query instanceof TimestampedWindowKeyWithHeadersQuery) {
result = runTimestampedWindowKeyWithHeadersQuery((TimestampedWindowKeyWithHeadersQuery<K, V>) query, positionBound, config);
+ handledLocally = true;
} else if (query instanceof WindowRangeQuery) {
result = runWindowRangeQuery((WindowRangeQuery<K, ValueTimestampHeaders<V>>) query, positionBound, config);
+ handledLocally = true;
+ } else if (query instanceof TimestampedWindowRangeWithHeadersQuery) {
+ result = runTimestampedWindowRangeWithHeadersQuery((TimestampedWindowRangeWithHeadersQuery<K, V>) query, positionBound, config);
+ handledLocally = true;
} else {
result = wrapped().query(query, positionBound, config);
+ handledLocally = false;
}
if (config.isCollectExecutionInfo()) {
- final String conversionType = isUnderlyingStoreTimestamped()
- ? "with conversion to ValueAndTimestamp"
- : "with extraction of plain values";
result.addExecutionInfo(
- "Handled in " + getClass() + " " + conversionType + " in "
- + (time.nanoseconds() - start) + "ns");
+ "Handled in " + getClass()
+ + (handledLocally ? " with serdes " + serdes : "")
+ + " in " + (time.nanoseconds() - start) + "ns");
}
return result;
}
@@ -250,8 +260,12 @@ public class MeteredTimestampedWindowStoreWithHeaders<K, V>
);
final QueryResult<WindowStoreIterator<byte[]>> rawResult = wrapped().query(rawKeyQuery, positionBound, config);
if (rawResult.isSuccess()) {
+ final K key = query.key();
final ReadOnlyRecordIterator<Windowed<K>, V> resultIterator =
- new MeteredWindowStoreWithHeadersReadOnlyRecordIterator(rawResult.getResult(), query.key());
+ new MeteredWindowStoreWithHeadersReadOnlyRecordIterator<Long>(
+ rawResult.getResult(),
+ (windowStart, headers) ->
+ new Windowed<>(key, new TimeWindow(windowStart, windowStart + windowSizeMs)));
queryResult = (QueryResult<R>) InternalQueryResultUtil.copyAndSubstituteDeserializedResult(rawResult, resultIterator);
} else {
queryResult = (QueryResult<R>) rawResult;
@@ -342,6 +356,48 @@ public class MeteredTimestampedWindowStoreWithHeaders<K, V>
return result;
}
+ /**
+ * Handles the {@code withWindowStartRange} form of {@link TimestampedWindowRangeWithHeadersQuery}
+ * by forwarding a raw byte-level {@link WindowRangeQuery#withWindowStartRange(java.time.Instant,
+ * java.time.Instant)} to the wrapped store and surfacing each entry as a {@link ReadOnlyRecord}
+ * (via {@link Record}) whose key is a {@link Windowed} of the deserialized key and the entry's
+ * window, carrying the stored value, event-time timestamp, and headers. The {@code withKey} form
+ * is rejected: it is handled by session stores, not window stores.
+ */
+ @SuppressWarnings("unchecked")
+ private <R> QueryResult<R> runTimestampedWindowRangeWithHeadersQuery(
+ final TimestampedWindowRangeWithHeadersQuery<K, V> query,
+ final PositionBound positionBound,
+ final QueryConfig config
+ ) {
+ final QueryResult<R> queryResult;
+
+ if (query.timeFrom().isPresent() && query.timeTo().isPresent()) {
+ final WindowRangeQuery<Bytes, byte[]> rawRangeQuery =
+ WindowRangeQuery.withWindowStartRange(query.timeFrom().get(), query.timeTo().get());
+ final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> rawResult = wrapped().query(rawRangeQuery, positionBound, config);
+ if (rawResult.isSuccess()) {
+ final ReadOnlyRecordIterator<Windowed<K>, V> resultIterator =
+ new MeteredWindowStoreWithHeadersReadOnlyRecordIterator<Windowed<Bytes>>(
+ rawResult.getResult(),
+ (windowedBytes, headers) ->
+ new Windowed<>(deserializeKey(windowedBytes.key().get(), headers), windowedBytes.window()));
+ queryResult = (QueryResult<R>) InternalQueryResultUtil.copyAndSubstituteDeserializedResult(rawResult, resultIterator);
+ } else {
+ queryResult = (QueryResult<R>) rawResult;
+ }
+ } else {
+ queryResult = QueryResult.forFailure(
+ FailureReason.UNKNOWN_QUERY_TYPE,
+ "This store (" + getClass() + ") doesn't know how to"
+ + " execute the given query (" + query + ") because"
+ + " WindowStores only supports TimestampedWindowRangeWithHeadersQuery.withWindowStartRange."
+ + " Contact the store maintainer if you need support for a new query type."
+ );
+ }
+ return queryResult;
+ }
+
@Override
public KeyValueIterator<Windowed<K>, ValueTimestampHeaders<V>> fetch(
final K keyFrom,
@@ -572,38 +628,53 @@ public class MeteredTimestampedWindowStoreWithHeaders<K, V>
}
/**
- * Iterator backing {@link TimestampedWindowKeyWithHeadersQuery}: yields each entry as a
+ * Iterator backing both headers-aware window query types: the single-key
+ * {@link TimestampedWindowKeyWithHeadersQuery} and the {@code withWindowStartRange} form of
+ * {@link TimestampedWindowRangeWithHeadersQuery}. Each entry is surfaced as a
* {@link ReadOnlyRecord} (implemented by {@link Record}) whose key is a {@link Windowed} of the
- * queried key and the entry's window, carrying the value, stored event-time timestamp, and the
- * stored headers, with the headers frozen so a caller cannot mutate the read-only result.
+ * entry's key and window, carrying the value, stored event-time timestamp, and the stored
+ * headers, with the headers frozen so a caller cannot mutate the read-only result.
+ *
+ * <p>The two query types differ only in how the {@link Windowed} key is derived from a raw
+ * entry, which the caller supplies as {@code toWindowedKey}: a single-key result yields the
+ * window-start timestamp ({@code RawKey} = {@link Long}) and reuses the queried key,
+ * reconstructing the window from {@code windowSizeMs}; a range result already carries the full
+ * {@code Windowed<Bytes>} ({@code RawKey} = {@code Windowed<Bytes>}), from which the key is
+ * deserialized and the window used directly.
*
* <p>A {@link ReadOnlyRecord} timestamp is contractually non-negative, so an entry with a
* negative stored timestamp cannot be represented -- for example a {@code WithHeaders} store
* built over a plain, non-timestamped window supplier surfaces entries with
- * {@code NO_TIMESTAMP} (-1). This mirrors the rule the point/range headers queries apply. But
- * because a lazily-evaluated iterator has already returned a successful {@link QueryResult}
- * before any entry is read, such an entry cannot be surfaced as a query-level failure; it is
- * instead reported by throwing a {@link StreamsException} while advancing the iterator.
+ * {@code NO_TIMESTAMP} (-1). This mirrors the rule the point/range headers queries apply. A
+ * value that deserializes to null has no event-time either and is reported the same way (though,
+ * unlike {@code NO_TIMESTAMP}, that is a fail-fast safety net -- a native store cannot hold a
+ * null value). But because a lazily-evaluated iterator has already returned a successful
+ * {@link QueryResult} before any entry is read, such an entry cannot be surfaced as a query-level
+ * failure; it is instead reported by throwing a {@link StreamsException} while advancing the
+ * iterator.
*
* <p>Because {@link #next()} can throw mid-iteration, the caller must always close this iterator
* (via try-with-resources or a {@code finally} block) even when iteration throws: a caller that
* catches the exception and abandons the iterator without closing it leaks the underlying store
* iterator and leaves the {@code num-open-iterators} metric permanently incremented.
+ *
+ * @param <RawKey> the raw iterator's key type: a {@link Long} window-start for a single-key
+ * result, or a {@code Windowed<Bytes>} for a range result
*/
- private class MeteredWindowStoreWithHeadersReadOnlyRecordIterator
+ private class MeteredWindowStoreWithHeadersReadOnlyRecordIterator<RawKey>
implements ReadOnlyRecordIterator<Windowed<K>, V>, MeteredIterator {
- private final WindowStoreIterator<byte[]> iter;
- private final K key;
+ private final KeyValueIterator<RawKey, byte[]> iter;
+ private final BiFunction<RawKey, Headers, Windowed<K>> toWindowedKey;
private final long startNs;
private final long startTimestampMs;
private MeteredWindowStoreWithHeadersReadOnlyRecordIterator(
- final WindowStoreIterator<byte[]> iter,
- final K key
+ final KeyValueIterator<RawKey, byte[]> iter,
+ final BiFunction<RawKey, Headers, Windowed<K>> toWindowedKey
) {
this.iter = iter;
- this.key = key;
+ this.toWindowedKey = toWindowedKey;
this.startNs = time.nanoseconds();
this.startTimestampMs = time.milliseconds();
numOpenIterators.increment();
@@ -622,18 +693,27 @@ public class MeteredTimestampedWindowStoreWithHeaders<K, V>
@Override
public ReadOnlyRecord<Windowed<K>, V> next() {
- final KeyValue<Long, byte[]> next = iter.next();
- final long windowStart = next.key;
+ final KeyValue<RawKey, byte[]> next = iter.next();
final ValueTimestampHeaders<V> valueTimestampHeaders = deserializeValue(next.value);
- final Headers headers = valueTimestampHeaders.headers();
+ // A ReadOnlyRecord carries the stored event-time, so an entry that has none cannot be
+ // represented. A value that deserializes to null has no event-time, so it is reported the
+ // same way as a negative/absent timestamp (below). The headers deref is guarded so we can
+ // build the error message rather than NPE. This is a fail-fast safety net: a null value is
+ // unreachable for a native store (a null value cannot be stored).
+ final Headers headers = valueTimestampHeaders != null ? valueTimestampHeaders.headers() : new RecordHeaders();
+ final Windowed<K> windowedKey = toWindowedKey.apply(next.key, headers);
+ if (valueTimestampHeaders == null) {
+ throw new StreamsException(
+ "Cannot represent the stored record for key [" + windowedKey.key()
+ + "] and window [" + windowedKey.window() + "] as a ReadOnlyRecord:"
+ + " its value is null.");
+ }
if (valueTimestampHeaders.timestamp() < 0) {
throw new StreamsException(
- "Cannot represent the stored record for key [" + key + "] and window start ["
- + windowStart + "] as a ReadOnlyRecord: its timestamp ("
- + valueTimestampHeaders.timestamp() + ") is negative.");
+ "Cannot represent the stored record for key [" + windowedKey.key()
+ + "] and window [" + windowedKey.window() + "] as a ReadOnlyRecord:"
+ + " its timestamp (" + valueTimestampHeaders.timestamp() + ") is negative.");
}
- final Windowed<K> windowedKey =
- new Windowed<>(key, new TimeWindow(windowStart, windowStart + windowSizeMs));
final Record<Windowed<K>, V> record = new Record<>(
windowedKey,
valueTimestampHeaders.value(),
diff --git a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeaders.java b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeaders.java
index 91fd584523d..dbc46b1c54d 100644
--- a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeaders.java
+++ b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeaders.java
@@ -16,20 +16,21 @@
*/
package org.apache.kafka.streams.state.internals;
-import org.apache.kafka.streams.processor.StateStore;
-import org.apache.kafka.streams.query.Position;
-import org.apache.kafka.streams.query.PositionBound;
-import org.apache.kafka.streams.query.Query;
-import org.apache.kafka.streams.query.QueryConfig;
-import org.apache.kafka.streams.query.QueryResult;
import org.apache.kafka.streams.state.HeadersBytesStore;
/**
* RocksDB-backed session store with support for record headers.
* <p>
- * This store extends {@link RocksDBSessionStore} and returns
- * {@link QueryResult#forUnknownQueryType(Query, StateStore)} for all queries,
- * as IQv2 query handling is done at the metered layer.
+ * This store extends {@link RocksDBSessionStore} and relies on its inherited IQv2 query handling
+ * (via {@code StoreQueryUtils}); header-aware value (de)serialization is performed at the metered
+ * layer.
+ * <p>
+ * There is deliberately no {@code query()} override to gate query types: the metered wrapper
+ * ({@code MeteredSessionStoreWithHeaders}) intercepts and deserializes every query that would
+ * otherwise be served from this store's raw header-format bytes, delegating downward only queries
+ * that {@code StoreQueryUtils} rejects as {@code UNKNOWN_QUERY_TYPE}. So raw header-format value
+ * bytes are never returned directly to a caller; teaching {@code StoreQueryUtils} to serve a new
+ * query type from session stores would require matching interception at the metered layer.
* <p>
* The storage format for values is: [headersSize(varint)][headersBytes][aggregationBytes]
*
@@ -40,25 +41,4 @@ class RocksDBSessionStoreWithHeaders extends RocksDBSessionStore implements Head
RocksDBSessionStoreWithHeaders(final AbstractRocksDBSegmentedBytesStore<?> bytesStore) {
super(bytesStore);
}
-
- @Override
- public <R> QueryResult<R> query(final Query<R> query,
- final PositionBound positionBound,
- final QueryConfig config) {
- final long start = config.isCollectExecutionInfo() ? System.nanoTime() : -1L;
- final QueryResult<R> result;
- final Position position = getPosition();
-
- synchronized (position) {
- result = QueryResult.forUnknownQueryType(query, this);
-
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + this.getClass() + " in " + (System.nanoTime() - start) + "ns"
- );
- }
- result.setPosition(position.copy());
- }
- return result;
- }
}
diff --git a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeaders.java b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeaders.java
index e4e517dc1e2..8faa11ca0a2 100644
--- a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeaders.java
+++ b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeaders.java
@@ -16,20 +16,21 @@
*/
package org.apache.kafka.streams.state.internals;
-import org.apache.kafka.streams.processor.StateStore;
-import org.apache.kafka.streams.query.Position;
-import org.apache.kafka.streams.query.PositionBound;
-import org.apache.kafka.streams.query.Query;
-import org.apache.kafka.streams.query.QueryConfig;
-import org.apache.kafka.streams.query.QueryResult;
import org.apache.kafka.streams.state.HeadersBytesStore;
/**
* RocksDB-backed time-ordered session store with support for record headers.
* <p>
- * This store extends {@link RocksDBTimeOrderedSessionStore} and returns
- * {@link QueryResult#forUnknownQueryType(Query, StateStore)} for all queries,
- * as IQv2 query handling is done at the metered layer.
+ * This store extends {@link RocksDBTimeOrderedSessionStore} and relies on its inherited IQv2 query
+ * handling (via {@code StoreQueryUtils}); header-aware value (de)serialization is performed at the
+ * metered layer.
+ * <p>
+ * There is deliberately no {@code query()} override to gate query types: the metered wrapper
+ * ({@code MeteredSessionStoreWithHeaders}) intercepts and deserializes every query that would
+ * otherwise be served from this store's raw header-format bytes, delegating downward only queries
+ * that {@code StoreQueryUtils} rejects as {@code UNKNOWN_QUERY_TYPE}. So raw header-format value
+ * bytes are never returned directly to a caller; teaching {@code StoreQueryUtils} to serve a new
+ * query type from session stores would require matching interception at the metered layer.
* <p>
* The storage format for values is: [headersSize(varint)][headersBytes][aggregationBytes]
*
@@ -40,25 +41,4 @@ class RocksDBTimeOrderedSessionStoreWithHeaders extends RocksDBTimeOrderedSessio
RocksDBTimeOrderedSessionStoreWithHeaders(final RocksDBTimeOrderedSessionSegmentedBytesStoreWithHeaders store) {
super(store);
}
-
- @Override
- public <R> QueryResult<R> query(final Query<R> query,
- final PositionBound positionBound,
- final QueryConfig config) {
- final long start = config.isCollectExecutionInfo() ? System.nanoTime() : -1L;
- final QueryResult<R> result;
- final Position position = getPosition();
-
- synchronized (position) {
- result = QueryResult.forUnknownQueryType(query, this);
-
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + this.getClass() + " in " + (System.nanoTime() - start) + "ns"
- );
- }
- result.setPosition(position.copy());
- }
- return result;
- }
}
diff --git a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeaders.java b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeaders.java
index 08a8f1771b7..803a85018c6 100644
--- a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeaders.java
+++ b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeaders.java
@@ -16,12 +16,6 @@
*/
package org.apache.kafka.streams.state.internals;
-import org.apache.kafka.streams.query.Position;
-import org.apache.kafka.streams.query.PositionBound;
-import org.apache.kafka.streams.query.Query;
-import org.apache.kafka.streams.query.QueryConfig;
-import org.apache.kafka.streams.query.QueryResult;
-import org.apache.kafka.streams.query.WindowKeyQuery;
import org.apache.kafka.streams.state.HeadersBytesStore;
import org.apache.kafka.streams.state.TimestampedBytesStore;
@@ -32,6 +26,15 @@ import org.apache.kafka.streams.state.TimestampedBytesStore;
* {@link TimestampedBytesStore} (for timestamp support) and {@link HeadersBytesStore}
* (for header support) marker interfaces.
* <p>
+ * IQv2 query handling is inherited (via {@code StoreQueryUtils}); header-aware value
+ * (de)serialization is performed at the metered layer. There is deliberately no {@code query()}
+ * override to gate query types: the metered wrapper ({@code MeteredTimestampedWindowStoreWithHeaders})
+ * intercepts and deserializes every query that would otherwise be served from this store's raw
+ * header-format bytes, delegating downward only queries that {@code StoreQueryUtils} rejects as
+ * {@code UNKNOWN_QUERY_TYPE}. So raw header-format value bytes are never returned directly to a
+ * caller; teaching {@code StoreQueryUtils} to serve a new query type from window stores would
+ * require matching interception at the metered layer.
+ * <p>
* The storage format for values is: [headersSize(varint)][headersBytes][timestamp(8)][value]
*
* @see RocksDBTimeOrderedWindowStore
@@ -45,34 +48,4 @@ class RocksDBTimeOrderedWindowStoreWithHeaders extends RocksDBTimeOrderedWindowS
final long windowSize) {
super(store, retainDuplicates, windowSize);
}
-
- @Override
- public <R> QueryResult<R> query(final Query<R> query,
- final PositionBound positionBound,
- final QueryConfig config) {
- // KIP-1356 (window key query only): the headers-aware TimestampedWindowKeyWithHeadersQuery
- // forwards a raw WindowKeyQuery to this native store, so enable WindowKeyQuery via the inherited
- // RocksDBTimeOrderedWindowStore handling (StoreQueryUtils). Every other query type (e.g.
- // WindowRangeQuery) stays UNKNOWN_QUERY_TYPE here, unchanged from before; those are enabled by
- // their own KIP-1356 follow-ups.
- if (query instanceof WindowKeyQuery) {
- return super.query(query, positionBound, config);
- }
-
- final long start = config.isCollectExecutionInfo() ? System.nanoTime() : -1L;
- final QueryResult<R> result;
- final Position position = getPosition();
-
- synchronized (position) {
- result = QueryResult.forUnknownQueryType(query, this);
-
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + this.getClass() + " in " + (System.nanoTime() - start) + "ns"
- );
- }
- result.setPosition(position.copy());
- }
- return result;
- }
}
diff --git a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeaders.java b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeaders.java
index 304975279f3..61c66232d80 100644
--- a/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeaders.java
+++ b/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeaders.java
@@ -16,12 +16,6 @@
*/
package org.apache.kafka.streams.state.internals;
-import org.apache.kafka.streams.query.Position;
-import org.apache.kafka.streams.query.PositionBound;
-import org.apache.kafka.streams.query.Query;
-import org.apache.kafka.streams.query.QueryConfig;
-import org.apache.kafka.streams.query.QueryResult;
-import org.apache.kafka.streams.query.WindowKeyQuery;
import org.apache.kafka.streams.state.HeadersBytesStore;
import org.apache.kafka.streams.state.TimestampedBytesStore;
@@ -32,6 +26,15 @@ import org.apache.kafka.streams.state.TimestampedBytesStore;
* {@link TimestampedBytesStore} (for timestamp support) and {@link HeadersBytesStore}
* (for header support) marker interfaces.
* <p>
+ * IQv2 query handling is inherited (via {@code StoreQueryUtils}); header-aware value
+ * (de)serialization is performed at the metered layer. There is deliberately no {@code query()}
+ * override to gate query types: the metered wrapper ({@code MeteredTimestampedWindowStoreWithHeaders})
+ * intercepts and deserializes every query that would otherwise be served from this store's raw
+ * header-format bytes, delegating downward only queries that {@code StoreQueryUtils} rejects as
+ * {@code UNKNOWN_QUERY_TYPE}. So raw header-format value bytes are never returned directly to a
+ * caller; teaching {@code StoreQueryUtils} to serve a new query type from window stores would
+ * require matching interception at the metered layer.
+ * <p>
* The storage format for values is: [headersSize(varint)][headersBytes][timestamp(8)][value]
* <p>
* This implementation uses segment-level versioning for backward compatibility:
@@ -53,34 +56,4 @@ class RocksDBTimestampedWindowStoreWithHeaders extends RocksDBWindowStore implem
final long windowSize) {
super(bytesStore, retainDuplicates, windowSize);
}
-
- @Override
- public <R> QueryResult<R> query(final Query<R> query,
- final PositionBound positionBound,
- final QueryConfig config) {
- // KIP-1356 (window key query only): the headers-aware TimestampedWindowKeyWithHeadersQuery
- // forwards a raw WindowKeyQuery to this native store, so enable WindowKeyQuery via the inherited
- // RocksDBWindowStore handling (StoreQueryUtils). Every other query type (e.g. WindowRangeQuery)
- // stays UNKNOWN_QUERY_TYPE here, unchanged from before; those are enabled by their own KIP-1356
- // follow-ups.
- if (query instanceof WindowKeyQuery) {
- return super.query(query, positionBound, config);
- }
-
- final long start = config.isCollectExecutionInfo() ? System.nanoTime() : -1L;
- final QueryResult<R> result;
- final Position position = getPosition();
-
- synchronized (position) {
- result = QueryResult.forUnknownQueryType(query, this);
-
- if (config.isCollectExecutionInfo()) {
- result.addExecutionInfo(
- "Handled in " + this.getClass() + " in " + (System.nanoTime() - start) + "ns"
- );
- }
- result.setPosition(position.copy());
- }
- return result;
- }
}
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeadersTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeadersTest.java
index 6b9d602cf8e..d57c9c02e2e 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeadersTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredSessionStoreWithHeadersTest.java
@@ -37,6 +37,7 @@ import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.internals.SessionWindow;
import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.processor.api.ReadOnlyRecord;
import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
import org.apache.kafka.streams.processor.internals.ProcessorRecordContext;
import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
@@ -45,9 +46,11 @@ import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.Query;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
import org.apache.kafka.streams.query.WindowRangeQuery;
import org.apache.kafka.streams.state.AggregationWithHeaders;
import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
import org.apache.kafka.streams.state.ReadOnlySessionStore;
import org.apache.kafka.streams.state.SessionStore;
import org.apache.kafka.test.KeyValueIteratorStub;
@@ -63,6 +66,7 @@ import org.mockito.quality.Strictness;
import java.util.Collections;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.stream.Collectors;
import static org.apache.kafka.common.utils.Utils.mkEntry;
@@ -797,6 +801,199 @@ public class MeteredSessionStoreWithHeadersTest {
assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getFailureReason());
}
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldHandleTimestampedWindowRangeWithHeadersQueryWithKey() {
+ setUp();
+ init();
+
+ final Headers headers = new RecordHeaders();
+ headers.add("key1", "value1".getBytes());
+ final AggregationWithHeaders<String> valueAndHeaders = AggregationWithHeaders.make(VALUE, headers);
+ final AggregationWithHeadersSerializer<String> serializer = new AggregationWithHeadersSerializer<>(Serdes.String().serializer());
+ final byte[] serializedValue = serializer.serialize(CHANGELOG_TOPIC, valueAndHeaders);
+
+ // A non-degenerate session window (unlike the file's default WINDOWED_KEY_BYTES, whose
+ // SessionWindow(0, 0) would make a timestamp-from-window-end assertion trivially true).
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new SessionWindow(START_TIMESTAMP, END_TIMESTAMP));
+ final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> rawResult =
+ QueryResult.forResult(new KeyValueIteratorStub<>(
+ Collections.singleton(KeyValue.pair(windowedKeyBytes, serializedValue)).iterator()));
+ when(innerStore.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) rawResult);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals(KEY, record.key().key());
+ assertEquals(START_TIMESTAMP, record.key().window().start());
+ assertEquals(END_TIMESTAMP, record.key().window().end());
+ assertEquals(VALUE, record.value());
+ // The timestamp is sourced from the session window's end, not any per-record field
+ // (AggregationWithHeaders carries no timestamp of its own).
+ assertEquals(END_TIMESTAMP, record.timestamp());
+ assertEquals(headers, record.headers());
+ // returned headers are a read-only snapshot: neither add nor remove is allowed
+ assertThrows(IllegalStateException.class, () -> record.headers().add("x", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("key1"));
+ assertFalse(iterator.hasNext());
+ }
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldTolerateNullValueForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ init();
+
+ // A value that deserializes to null must not NPE: the record surfaces a null value and empty
+ // headers, matching the sibling MeteredSessionStoreWithHeadersIterator.next().
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new SessionWindow(START_TIMESTAMP, END_TIMESTAMP));
+ when(innerStore.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(new KeyValueIteratorStub<>(
+ Collections.singleton(KeyValue.pair(windowedKeyBytes, (byte[]) null)).iterator())));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals(KEY, record.key().key());
+ assertNull(record.value());
+ assertEquals(END_TIMESTAMP, record.timestamp());
+ assertEquals(new RecordHeaders(), record.headers());
+ assertFalse(iterator.hasNext());
+ }
+ }
+
+ @Test
+ public void shouldRejectWithWindowStartRangeFormForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ init();
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ java.time.Instant.ofEpochMilli(0L), java.time.Instant.ofEpochMilli(0L)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertTrue(result.isFailure());
+ assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getFailureReason());
+ assertTrue(
+ result.getFailureMessage().contains("SessionStores only support TimestampedWindowRangeWithHeadersQuery.withKey"),
+ "unexpected message: " + result.getFailureMessage());
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldForwardRawKeyQueryForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ init();
+
+ when(innerStore.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(
+ new KeyValueIteratorStub<>(Collections.<KeyValue<Windowed<Bytes>, byte[]>>emptyList().iterator())));
+
+ // Close the iterator the query opens so num-open-iterators returns to 0.
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false)).getResult()) {
+ // The typed withKey query is translated into a raw byte-level WindowRangeQuery.withKey carrying
+ // the serialized key (and no window-start range), forwarded to the wrapped store.
+ final ArgumentCaptor<WindowRangeQuery> captor = ArgumentCaptor.forClass(WindowRangeQuery.class);
+ verify(innerStore).query(captor.capture(), any(PositionBound.class), any(QueryConfig.class));
+ final WindowRangeQuery<?, ?> rawQuery = captor.getValue();
+ assertEquals(Optional.of(KEY_BYTES), rawQuery.getKey());
+ assertEquals(Optional.empty(), rawQuery.getTimeFrom());
+ assertEquals(Optional.empty(), rawQuery.getTimeTo());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldPropagateWrappedStoreFailureForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ init();
+ when(innerStore.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forFailure(FailureReason.STORE_EXCEPTION, "boom"));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertFalse(result.isSuccess());
+ assertEquals(FailureReason.STORE_EXCEPTION, result.getFailureReason());
+ assertEquals("boom", result.getFailureMessage());
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldTrackNumOpenIteratorsForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ init();
+
+ final AggregationWithHeadersSerializer<String> serializer = new AggregationWithHeadersSerializer<>(Serdes.String().serializer());
+ final byte[] serializedValue = serializer.serialize(CHANGELOG_TOPIC, AggregationWithHeaders.make(VALUE, new RecordHeaders()));
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new SessionWindow(START_TIMESTAMP, END_TIMESTAMP));
+ when(innerStore.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(new KeyValueIteratorStub<>(
+ Collections.singleton(KeyValue.pair(windowedKeyBytes, serializedValue)).iterator())));
+
+ final KafkaMetric openIteratorsMetric = metric("num-open-iterators");
+ assertThat((Long) openIteratorsMetric.metricValue(), equalTo(0L));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+ assertTrue(result.isSuccess());
+
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertThat((Long) openIteratorsMetric.metricValue(), equalTo(1L));
+ }
+ assertThat((Long) openIteratorsMetric.metricValue(), equalTo(0L));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldDecrementOpenIteratorsTwiceWhenClosedTwiceForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ init();
+
+ final AggregationWithHeadersSerializer<String> serializer = new AggregationWithHeadersSerializer<>(Serdes.String().serializer());
+ final byte[] serializedValue = serializer.serialize(CHANGELOG_TOPIC, AggregationWithHeaders.make(VALUE, new RecordHeaders()));
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new SessionWindow(START_TIMESTAMP, END_TIMESTAMP));
+ when(innerStore.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(new KeyValueIteratorStub<>(
+ Collections.singleton(KeyValue.pair(windowedKeyBytes, serializedValue)).iterator())));
+
+ final KafkaMetric openIteratorsMetric = metric("num-open-iterators");
+ final ReadOnlyRecordIterator<Windowed<String>, String> iterator = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false)).getResult();
+
+ assertThat((Long) openIteratorsMetric.metricValue(), equalTo(1L));
+ iterator.close();
+ assertThat((Long) openIteratorsMetric.metricValue(), equalTo(0L));
+ // close() is intentionally not idempotent (matching the sibling metered iterators): each call
+ // decrements, so a repeated close drives the gauge below zero. Callers must close exactly once.
+ iterator.close();
+ assertThat((Long) openIteratorsMetric.metricValue(), equalTo(-1L));
+ }
+
@SuppressWarnings("unchecked")
@Test
public void shouldDelegateUnknownQueryToWrappedStore() {
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeadersTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeadersTest.java
index 3aa0a1b87ff..8820319f0e3 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeadersTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/MeteredTimestampedWindowStoreWithHeadersTest.java
@@ -37,6 +37,7 @@ import org.apache.kafka.streams.errors.StreamsException;
import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.kstream.internals.TimeWindow;
import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.processor.api.ReadOnlyRecord;
import org.apache.kafka.streams.processor.internals.ProcessorRecordContext;
import org.apache.kafka.streams.processor.internals.ProcessorStateManager;
import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
@@ -46,7 +47,9 @@ import org.apache.kafka.streams.query.Query;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
import org.apache.kafka.streams.query.TimestampedWindowKeyWithHeadersQuery;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
import org.apache.kafka.streams.query.WindowKeyQuery;
+import org.apache.kafka.streams.query.WindowRangeQuery;
import org.apache.kafka.streams.state.KeyValueIterator;
import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
import org.apache.kafka.streams.state.ReadOnlyWindowStore;
@@ -573,7 +576,7 @@ public class MeteredTimestampedWindowStoreWithHeadersTest {
setUp();
store.init(context, store);
when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
- .thenReturn((QueryResult) QueryResult.forResult(windowStoreIterator(List.of())));
+ .thenReturn((QueryResult) QueryResult.forResult(windowKeyIterator(List.of())));
final KafkaMetric openIterators = numOpenIteratorsMetric();
assertEquals(0L, (Long) openIterators.metricValue());
@@ -598,7 +601,7 @@ public class MeteredTimestampedWindowStoreWithHeadersTest {
setUp();
store.init(context, store);
when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
- .thenReturn((QueryResult) QueryResult.forResult(windowStoreIterator(List.of())));
+ .thenReturn((QueryResult) QueryResult.forResult(windowKeyIterator(List.of())));
final KafkaMetric openIterators = numOpenIteratorsMetric();
final ReadOnlyRecordIterator<Windowed<String>, String> iterator = store.query(
@@ -626,7 +629,7 @@ public class MeteredTimestampedWindowStoreWithHeadersTest {
.serialize("topic", ValueTimestampHeaders.make("value", -1L, HEADERS));
when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
.thenReturn((QueryResult) QueryResult.forResult(
- windowStoreIterator(List.of(KeyValue.pair(5L, negativeTimestampBytes)))));
+ windowKeyIterator(List.of(KeyValue.pair(5L, negativeTimestampBytes)))));
final KafkaMetric openIterators = numOpenIteratorsMetric();
final ReadOnlyRecordIterator<Windowed<String>, String> iterator = store.query(
@@ -647,14 +650,15 @@ public class MeteredTimestampedWindowStoreWithHeadersTest {
@SuppressWarnings({"unchecked", "rawtypes"})
private WindowKeyQuery<?, ?> forwardedRawWindowKeyQuery(final Query<?> query) {
when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
- .thenReturn((QueryResult) QueryResult.forResult(windowStoreIterator(List.of())));
- store.query(query, PositionBound.unbounded(), new QueryConfig(false));
+ .thenReturn((QueryResult) QueryResult.forResult(windowKeyIterator(List.of())));
+ // Close the iterator the query opens so num-open-iterators returns to 0.
+ ((ReadOnlyRecordIterator<?, ?>) store.query(query, PositionBound.unbounded(), new QueryConfig(false)).getResult()).close();
final ArgumentCaptor<WindowKeyQuery> captor = ArgumentCaptor.forClass(WindowKeyQuery.class);
verify(innerStoreMock).query(captor.capture(), any(PositionBound.class), any(QueryConfig.class));
return captor.getValue();
}
- private static WindowStoreIterator<byte[]> windowStoreIterator(final List<KeyValue<Long, byte[]>> data) {
+ private static WindowStoreIterator<byte[]> windowKeyIterator(final List<KeyValue<Long, byte[]>> data) {
final Iterator<KeyValue<Long, byte[]>> iterator = data.iterator();
return new WindowStoreIterator<>() {
@Override
@@ -677,6 +681,213 @@ public class MeteredTimestampedWindowStoreWithHeadersTest {
};
}
+ @Test
+ public void shouldPropagateWindowRangeBoundsForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+
+ final Instant timeFrom = Instant.ofEpochMilli(5);
+ final Instant timeTo = Instant.ofEpochMilli(100);
+ final WindowRangeQuery<?, ?> rawQuery = forwardedRawRangeQuery(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(timeFrom, timeTo));
+
+ // The typed query is translated into a raw byte-level WindowRangeQuery.withWindowStartRange
+ // with the same window-start range, forwarded to the wrapped store.
+ assertEquals(Optional.empty(), rawQuery.getKey());
+ assertEquals(Optional.of(timeFrom), rawQuery.getTimeFrom());
+ assertEquals(Optional.of(timeTo), rawQuery.getTimeTo());
+ }
+
+ @Test
+ public void shouldRejectWithKeyFormForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey(KEY),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertFalse(result.isSuccess());
+ assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getFailureReason());
+ assertTrue(
+ result.getFailureMessage().contains("WindowStores only supports TimestampedWindowRangeWithHeadersQuery.withWindowStartRange"),
+ "unexpected message: " + result.getFailureMessage());
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldPropagateWrappedStoreFailureForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forFailure(FailureReason.STORE_EXCEPTION, "boom"));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(5), Instant.ofEpochMilli(100)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertFalse(result.isSuccess());
+ assertEquals(FailureReason.STORE_EXCEPTION, result.getFailureReason());
+ assertEquals("boom", result.getFailureMessage());
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldTrackNumOpenIteratorsForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(windowRangeIterator(List.of())));
+
+ final KafkaMetric openIterators = numOpenIteratorsMetric();
+ assertEquals(0L, (Long) openIterators.metricValue());
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(5), Instant.ofEpochMilli(100)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+ assertTrue(result.isSuccess());
+
+ // The query's ReadOnlyRecordIterator registers itself on open and deregisters on close.
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertEquals(1L, (Long) openIterators.metricValue());
+ }
+ assertEquals(0L, (Long) openIterators.metricValue());
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldDecrementOpenIteratorsTwiceWhenClosedTwiceForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(windowRangeIterator(List.of())));
+
+ final KafkaMetric openIterators = numOpenIteratorsMetric();
+ final ReadOnlyRecordIterator<Windowed<String>, String> iterator = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(5), Instant.ofEpochMilli(100)),
+ PositionBound.unbounded(),
+ new QueryConfig(false)).getResult();
+
+ assertEquals(1L, (Long) openIterators.metricValue());
+ iterator.close();
+ assertEquals(0L, (Long) openIterators.metricValue());
+ // close() is intentionally not idempotent (matching the sibling metered iterators): each call
+ // decrements, so a repeated close drives the gauge below zero. Callers must close exactly once.
+ iterator.close();
+ assertEquals(-1L, (Long) openIterators.metricValue());
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldLeaveIteratorOpenWhenNextThrowsAndNotClosedForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+ // A stored entry with a negative timestamp cannot be represented as a ReadOnlyRecord, so next() throws.
+ final byte[] negativeTimestampBytes = new ValueTimestampHeadersSerializer<>(new StringSerializer())
+ .serialize("topic", ValueTimestampHeaders.make("value", -1L, HEADERS));
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new TimeWindow(5L, 5L + WINDOW_SIZE_MS));
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(
+ windowRangeIterator(List.of(KeyValue.pair(windowedKeyBytes, negativeTimestampBytes)))));
+
+ final KafkaMetric openIterators = numOpenIteratorsMetric();
+ final ReadOnlyRecordIterator<Windowed<String>, String> iterator = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(5), Instant.ofEpochMilli(100)),
+ PositionBound.unbounded(),
+ new QueryConfig(false)).getResult();
+
+ assertEquals(1L, (Long) openIterators.metricValue());
+ assertThrows(StreamsException.class, iterator::next);
+ assertEquals(1L, (Long) openIterators.metricValue());
+ iterator.close();
+ assertEquals(0L, (Long) openIterators.metricValue());
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldUseWindowFromRawResultForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+ // Deliberately a window whose length differs from WINDOW_SIZE_MS, proving the returned
+ // Windowed<K> comes straight from the raw range result rather than being reconstructed from
+ // windowSizeMs (unlike the single-key point query, which only gets a window-start Long back).
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new TimeWindow(1_000L, 5_000L));
+ final KeyValue<Windowed<Bytes>, byte[]> testData = KeyValue.pair(windowedKeyBytes, VALUE_TIMESTAMP_HEADERS_BYTES);
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(windowRangeIterator(List.of(testData))));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(0), Instant.ofEpochMilli(10_000)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals(KEY, record.key().key());
+ assertEquals(1_000L, record.key().window().start());
+ assertEquals(5_000L, record.key().window().end());
+ assertEquals("value", record.value());
+ assertEquals(TIMESTAMP, record.timestamp());
+ assertEquals(HEADERS, record.headers());
+ // returned headers are a read-only snapshot: neither add nor remove is allowed
+ assertThrows(IllegalStateException.class, () -> record.headers().add("x", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("header-key"));
+ assertFalse(iterator.hasNext());
+ }
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ @Test
+ public void shouldThrowForNullStoredValueForTimestampedWindowRangeWithHeadersQuery() {
+ setUp();
+ store.init(context, store);
+
+ // A ReadOnlyRecord carries the stored event-time; a value that deserializes to null has none,
+ // so the entry cannot be represented and next() throws (rather than NPE-ing on the null value).
+ final Windowed<Bytes> windowedKeyBytes = new Windowed<>(KEY_BYTES, new TimeWindow(1_000L, 5_000L));
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(
+ windowRangeIterator(List.of(KeyValue.pair(windowedKeyBytes, (byte[]) null)))));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(0), Instant.ofEpochMilli(10_000)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final StreamsException exception = assertThrows(StreamsException.class, iterator::next);
+ assertTrue(exception.getMessage().contains("its value is null"), exception.getMessage());
+ }
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private WindowRangeQuery<?, ?> forwardedRawRangeQuery(final Query<?> query) {
+ when(innerStoreMock.query(any(), any(PositionBound.class), any(QueryConfig.class)))
+ .thenReturn((QueryResult) QueryResult.forResult(windowRangeIterator(List.of())));
+ // Close the iterator the query opens so num-open-iterators returns to 0.
+ ((ReadOnlyRecordIterator<?, ?>) store.query(query, PositionBound.unbounded(), new QueryConfig(false)).getResult()).close();
+ final ArgumentCaptor<WindowRangeQuery> captor = ArgumentCaptor.forClass(WindowRangeQuery.class);
+ verify(innerStoreMock).query(captor.capture(), any(PositionBound.class), any(QueryConfig.class));
+ return captor.getValue();
+ }
+
+ private static KeyValueIterator<Windowed<Bytes>, byte[]> windowRangeIterator(final List<KeyValue<Windowed<Bytes>, byte[]>> data) {
+ return new KeyValueIteratorStub<>(data.iterator());
+ }
+
private KafkaMetric numOpenIteratorsMetric() {
return metrics.metrics().entrySet().stream()
.filter(entry -> entry.getKey().name().equals("num-open-iterators"))
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeadersTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeadersTest.java
index 2e04ab05852..8c140dc49d2 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeadersTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBSessionStoreWithHeadersTest.java
@@ -18,9 +18,10 @@ package org.apache.kafka.streams.state.internals;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.Windowed;
-import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.kstream.internals.SessionWindow;
import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
@@ -37,6 +38,7 @@ import org.junit.jupiter.api.Test;
import java.io.File;
import java.util.Properties;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -83,15 +85,29 @@ public class RocksDBSessionStoreWithHeadersTest {
}
@Test
- public void shouldReturnUnknownQueryTypeForWindowRangeQuery() {
- final WindowRangeQuery<Bytes, byte[]> query = WindowRangeQuery.withKey(
- new Bytes("test-key".getBytes())
- );
+ public void shouldHandleWindowRangeQuery() {
+ // KIP-1356: the withKey form of the headers-aware TimestampedWindowRangeWithHeadersQuery
+ // forwards a raw WindowRangeQuery to this native store, so enable WindowRangeQuery via the
+ // inherited RocksDBSessionStore handling (StoreQueryUtils), returning the raw stored
+ // header-format bytes (previously UNKNOWN_QUERY_TYPE for every query). This also fixes the
+ // same pre-existing gap for the plain (non-headers) WindowRangeQuery.withKey.
+ final Bytes key = new Bytes("test-key".getBytes());
+ final byte[] storedBytes = "headers+aggregation".getBytes();
+ final Windowed<Bytes> windowedKey = new Windowed<>(key, new SessionWindow(0L, 1_000L));
+ sessionStore.put(windowedKey, storedBytes);
+
+ final WindowRangeQuery<Bytes, byte[]> query = WindowRangeQuery.withKey(key);
final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
sessionStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- assertFalse(result.isSuccess());
- assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getFailureReason());
+ assertTrue(result.isSuccess(), "Expected WindowRangeQuery to succeed");
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext(), "Expected the stored session in the window range result");
+ final KeyValue<Windowed<Bytes>, byte[]> keyValue = iterator.next();
+ assertEquals(key, keyValue.key.key());
+ assertArrayEquals(storedBytes, keyValue.value, "Expected the raw stored bytes to be returned");
+ assertFalse(iterator.hasNext(), "Expected exactly one entry in the window range result");
+ }
assertNotNull(result.getPosition());
}
@@ -103,10 +119,13 @@ public class RocksDBSessionStoreWithHeadersTest {
final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
sessionStore.query(query, PositionBound.unbounded(), new QueryConfig(true));
- assertFalse(result.getExecutionInfo().isEmpty());
- assertTrue(result.getExecutionInfo().get(0).contains("Handled in"));
- assertTrue(result.getExecutionInfo().get(0).contains(
- RocksDBSessionStoreWithHeaders.class.getName()));
+ // The query now succeeds and returns an open store iterator, so close it to avoid a leak.
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertFalse(result.getExecutionInfo().isEmpty());
+ assertTrue(result.getExecutionInfo().get(0).contains("Handled in"));
+ assertTrue(result.getExecutionInfo().get(0).contains(
+ RocksDBSessionStoreWithHeaders.class.getName()));
+ }
}
@Test
@@ -117,6 +136,8 @@ public class RocksDBSessionStoreWithHeadersTest {
final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
sessionStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- assertTrue(result.getExecutionInfo().isEmpty());
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertTrue(result.getExecutionInfo().isEmpty());
+ }
}
}
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeadersTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeadersTest.java
index d9647494a66..ae0687c808a 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeadersTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedSessionStoreWithHeadersTest.java
@@ -18,9 +18,10 @@ package org.apache.kafka.streams.state.internals;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.Windowed;
-import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.kstream.internals.SessionWindow;
import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
@@ -37,6 +38,7 @@ import org.junit.jupiter.api.Test;
import java.io.File;
import java.util.Properties;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
@@ -83,15 +85,29 @@ public class RocksDBTimeOrderedSessionStoreWithHeadersTest {
}
@Test
- public void shouldReturnUnknownQueryTypeForWindowRangeQuery() {
- final WindowRangeQuery<Bytes, byte[]> query = WindowRangeQuery.withKey(
- new Bytes("test-key".getBytes())
- );
+ public void shouldHandleWindowRangeQuery() {
+ // KIP-1356: the withKey form of the headers-aware TimestampedWindowRangeWithHeadersQuery
+ // forwards a raw WindowRangeQuery to this native store, so enable WindowRangeQuery via the
+ // inherited RocksDBTimeOrderedSessionStore handling (StoreQueryUtils), returning the raw
+ // stored header-format bytes (previously UNKNOWN_QUERY_TYPE for every query). This also fixes
+ // the same pre-existing gap for the plain (non-headers) WindowRangeQuery.withKey.
+ final Bytes key = new Bytes("test-key".getBytes());
+ final byte[] storedBytes = "headers+aggregation".getBytes();
+ final Windowed<Bytes> windowedKey = new Windowed<>(key, new SessionWindow(0L, 1_000L));
+ sessionStore.put(windowedKey, storedBytes);
+
+ final WindowRangeQuery<Bytes, byte[]> query = WindowRangeQuery.withKey(key);
final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
sessionStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- assertFalse(result.isSuccess());
- assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getFailureReason());
+ assertTrue(result.isSuccess(), "Expected WindowRangeQuery to succeed");
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext(), "Expected the stored session in the window range result");
+ final KeyValue<Windowed<Bytes>, byte[]> keyValue = iterator.next();
+ assertEquals(key, keyValue.key.key());
+ assertArrayEquals(storedBytes, keyValue.value, "Expected the raw stored bytes to be returned");
+ assertFalse(iterator.hasNext(), "Expected exactly one entry in the window range result");
+ }
assertNotNull(result.getPosition());
}
@@ -103,10 +119,13 @@ public class RocksDBTimeOrderedSessionStoreWithHeadersTest {
final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
sessionStore.query(query, PositionBound.unbounded(), new QueryConfig(true));
- assertFalse(result.getExecutionInfo().isEmpty());
- assertTrue(result.getExecutionInfo().get(0).contains("Handled in"));
- assertTrue(result.getExecutionInfo().get(0).contains(
- RocksDBTimeOrderedSessionStoreWithHeaders.class.getName()));
+ // The query now succeeds and returns an open store iterator, so close it to avoid a leak.
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertFalse(result.getExecutionInfo().isEmpty());
+ assertTrue(result.getExecutionInfo().get(0).contains("Handled in"));
+ assertTrue(result.getExecutionInfo().get(0).contains(
+ RocksDBTimeOrderedSessionStoreWithHeaders.class.getName()));
+ }
}
@Test
@@ -117,6 +136,8 @@ public class RocksDBTimeOrderedSessionStoreWithHeadersTest {
final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
sessionStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- assertTrue(result.getExecutionInfo().isEmpty());
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertTrue(result.getExecutionInfo().isEmpty());
+ }
}
}
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeadersTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeadersTest.java
index 6b997c668b0..5b8f9b78bfd 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeadersTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeadersTest.java
@@ -20,7 +20,7 @@ import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
-import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
@@ -119,16 +119,34 @@ public class RocksDBTimeOrderedWindowStoreWithHeadersTest {
}
@Test
- public void shouldReturnUnknownQueryTypeForWindowRangeQuery() {
+ public void shouldHandleWindowRangeQuery() {
+ // KIP-1356: the withWindowStartRange form of the headers-aware
+ // TimestampedWindowRangeWithHeadersQuery forwards a raw WindowRangeQuery to this native store,
+ // so enable WindowRangeQuery via the inherited RocksDBTimeOrderedWindowStore handling
+ // (StoreQueryUtils), returning the raw stored header-format bytes (previously
+ // UNKNOWN_QUERY_TYPE); the metered store does the header-aware deserialization. This matches
+ // the adapter build path, and also fixes the same pre-existing gap for the plain (non-headers)
+ // WindowRangeQuery.
+ final Bytes key = new Bytes("test-key".getBytes());
+ final byte[] storedBytes = "headers+timestamp+value".getBytes();
+ final long windowStart = 1_000L;
+ windowStore.put(key, storedBytes, windowStart);
+
final WindowRangeQuery<Bytes, byte[]> query = WindowRangeQuery.withWindowStartRange(
Instant.ofEpochMilli(0),
- Instant.ofEpochMilli(Long.MAX_VALUE)
+ Instant.ofEpochMilli(RETENTION_PERIOD)
);
- final QueryResult<KeyValueIterator<org.apache.kafka.streams.kstream.Windowed<Bytes>, byte[]>> result =
+ final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
windowStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- assertFalse(result.isSuccess());
- assertEquals(FailureReason.UNKNOWN_QUERY_TYPE, result.getFailureReason());
+ assertTrue(result.isSuccess(), "Expected WindowRangeQuery to succeed");
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext(), "Expected the stored entry in the window range result");
+ final KeyValue<Windowed<Bytes>, byte[]> keyValue = iterator.next();
+ assertEquals(key, keyValue.key.key());
+ assertArrayEquals(storedBytes, keyValue.value, "Expected the raw stored bytes to be returned");
+ assertFalse(iterator.hasNext(), "Expected exactly one entry in the window range result");
+ }
assertNotNull(result.getPosition());
}
@@ -142,10 +160,13 @@ public class RocksDBTimeOrderedWindowStoreWithHeadersTest {
final QueryResult<WindowStoreIterator<byte[]>> result =
windowStore.query(query, PositionBound.unbounded(), new QueryConfig(true));
- assertFalse(result.getExecutionInfo().isEmpty());
- assertTrue(result.getExecutionInfo().get(0).contains("Handled in"));
- assertTrue(result.getExecutionInfo().get(0).contains(
- RocksDBTimeOrderedWindowStoreWithHeaders.class.getName()));
+ // The query succeeds and returns an open store iterator, so close it to avoid a leak.
+ try (WindowStoreIterator<byte[]> iterator = result.getResult()) {
+ assertFalse(result.getExecutionInfo().isEmpty());
+ assertTrue(result.getExecutionInfo().get(0).contains("Handled in"));
+ assertTrue(result.getExecutionInfo().get(0).contains(
+ RocksDBTimeOrderedWindowStoreWithHeaders.class.getName()));
+ }
}
@Test
@@ -158,6 +179,8 @@ public class RocksDBTimeOrderedWindowStoreWithHeadersTest {
final QueryResult<WindowStoreIterator<byte[]>> result =
windowStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- assertTrue(result.getExecutionInfo().isEmpty());
+ try (WindowStoreIterator<byte[]> iterator = result.getResult()) {
+ assertTrue(result.getExecutionInfo().isEmpty());
+ }
}
}
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeadersTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeadersTest.java
index 6d77fb3dd0e..3507a0fef80 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeadersTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedWindowStoreWithHeadersTest.java
@@ -20,7 +20,7 @@ import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsConfig;
-import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.kstream.Windowed;
import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
@@ -123,24 +123,33 @@ public class RocksDBTimestampedWindowStoreWithHeadersTest {
}
@Test
- public void shouldReturnUnknownQueryTypeForWindowRangeQuery() {
+ public void shouldHandleWindowRangeQuery() {
+ // KIP-1356: the withWindowStartRange form of the headers-aware
+ // TimestampedWindowRangeWithHeadersQuery forwards a raw WindowRangeQuery to this native store,
+ // so enable WindowRangeQuery via the inherited RocksDBWindowStore handling (StoreQueryUtils),
+ // returning the raw stored header-format bytes (previously UNKNOWN_QUERY_TYPE); the metered
+ // store does the header-aware deserialization. This matches the adapter build path, and also
+ // fixes the same pre-existing gap for the plain (non-headers) WindowRangeQuery.
+ final Bytes key = new Bytes("test-key".getBytes());
+ final byte[] storedBytes = "headers+timestamp+value".getBytes();
+ final long windowStart = 1_000L;
+ windowStore.put(key, storedBytes, windowStart);
+
final WindowRangeQuery<Bytes, byte[]> query = WindowRangeQuery.withWindowStartRange(
Instant.ofEpochMilli(0),
- Instant.ofEpochMilli(Long.MAX_VALUE)
+ Instant.ofEpochMilli(RETENTION_PERIOD)
);
- final PositionBound positionBound = PositionBound.unbounded();
- final QueryConfig config = new QueryConfig(false);
-
- final QueryResult<KeyValueIterator<org.apache.kafka.streams.kstream.Windowed<Bytes>, byte[]>> result =
- windowStore.query(query, positionBound, config);
+ final QueryResult<KeyValueIterator<Windowed<Bytes>, byte[]>> result =
+ windowStore.query(query, PositionBound.unbounded(), new QueryConfig(false));
- // Verify: Window store with headers currently returns UNKNOWN_QUERY_TYPE
- assertFalse(result.isSuccess(), "Expected query to fail with unknown query type");
- assertEquals(
- FailureReason.UNKNOWN_QUERY_TYPE,
- result.getFailureReason(),
- "Expected UNKNOWN_QUERY_TYPE failure reason"
- );
+ assertTrue(result.isSuccess(), "Expected WindowRangeQuery to succeed");
+ try (KeyValueIterator<Windowed<Bytes>, byte[]> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext(), "Expected the stored entry in the window range result");
+ final KeyValue<Windowed<Bytes>, byte[]> keyValue = iterator.next();
+ assertEquals(key, keyValue.key.key());
+ assertArrayEquals(storedBytes, keyValue.value, "Expected the raw stored bytes to be returned");
+ assertFalse(iterator.hasNext(), "Expected exactly one entry in the window range result");
+ }
assertNotNull(result.getPosition(), "Expected position to be set");
}
@@ -156,16 +165,19 @@ public class RocksDBTimestampedWindowStoreWithHeadersTest {
final QueryResult<WindowStoreIterator<byte[]>> result = windowStore.query(query, positionBound, config);
- // Verify: Execution info was collected
- assertFalse(result.getExecutionInfo().isEmpty(), "Expected execution info to be collected");
- assertTrue(
- result.getExecutionInfo().get(0).contains("Handled in"),
- "Expected execution info to contain handling information"
- );
- assertTrue(
- result.getExecutionInfo().get(0).contains(RocksDBTimestampedWindowStoreWithHeaders.class.getName()),
- "Expected execution info to mention the class name"
- );
+ // The query succeeds and returns an open store iterator, so close it to avoid a leak.
+ try (WindowStoreIterator<byte[]> iterator = result.getResult()) {
+ // Verify: Execution info was collected
+ assertFalse(result.getExecutionInfo().isEmpty(), "Expected execution info to be collected");
+ assertTrue(
+ result.getExecutionInfo().get(0).contains("Handled in"),
+ "Expected execution info to contain handling information"
+ );
+ assertTrue(
+ result.getExecutionInfo().get(0).contains(RocksDBTimestampedWindowStoreWithHeaders.class.getName()),
+ "Expected execution info to mention the class name"
+ );
+ }
}
@Test
@@ -180,7 +192,10 @@ public class RocksDBTimestampedWindowStoreWithHeadersTest {
final QueryResult<WindowStoreIterator<byte[]>> result = windowStore.query(query, positionBound, config);
- // Verify: No execution info was collected
- assertTrue(result.getExecutionInfo().isEmpty(), "Expected no execution info to be collected");
+ // The query succeeds and returns an open store iterator, so close it to avoid a leak.
+ try (WindowStoreIterator<byte[]> iterator = result.getResult()) {
+ // Verify: No execution info was collected
+ assertTrue(result.getExecutionInfo().isEmpty(), "Expected no execution info to be collected");
+ }
}
}
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/SessionStoreWithHeadersBuilderTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/SessionStoreWithHeadersBuilderTest.java
new file mode 100644
index 00000000000..84885d4ff4f
--- /dev/null
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/SessionStoreWithHeadersBuilderTest.java
@@ -0,0 +1,505 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.common.utils.internals.LogContext;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.SessionWindow;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.api.ReadOnlyRecord;
+import org.apache.kafka.streams.processor.internals.MockStreamsMetrics;
+import org.apache.kafka.streams.processor.internals.ProcessorRecordContext;
+import org.apache.kafka.streams.query.PositionBound;
+import org.apache.kafka.streams.query.QueryConfig;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
+import org.apache.kafka.streams.query.WindowRangeQuery;
+import org.apache.kafka.streams.state.AggregationWithHeaders;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.SessionStoreWithHeaders;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.mockito.junit.jupiter.MockitoSettings;
+import org.mockito.quality.Strictness;
+
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.lenient;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+@MockitoSettings(strictness = Strictness.STRICT_STUBS)
+public class SessionStoreWithHeadersBuilderTest {
+
+ private enum StoreType { NATIVE, ADAPTER, IN_MEMORY }
+
+ @Nested
+ class BuilderTests {
+ private static final String STORE_NAME = "name";
+ private static final String METRICS_SCOPE = "metricsScope";
+
+ @Mock
+ private SessionBytesStoreSupplier supplier;
+ @Mock
+ private RocksDBSessionStoreWithHeaders sessionStoreWithHeaders;
+
+ private SessionStoreWithHeadersBuilder<String, String> builder;
+
+ public void setUp() {
+ when(supplier.name()).thenReturn(STORE_NAME);
+ when(supplier.metricsScope()).thenReturn(METRICS_SCOPE);
+ when(supplier.get()).thenReturn(sessionStoreWithHeaders);
+
+ builder = new SessionStoreWithHeadersBuilder<>(
+ supplier,
+ Serdes.String(),
+ Serdes.String(),
+ new MockTime());
+ }
+
+ @Test
+ public void shouldHaveMeteredStoreAsOuterStore() {
+ setUp();
+ final SessionStoreWithHeaders<String, String> store = builder.build();
+ assertInstanceOf(MeteredSessionStoreWithHeaders.class, store);
+ }
+
+ @Test
+ public void shouldHaveChangeLoggingStoreByDefault() {
+ setUp();
+ final SessionStoreWithHeaders<String, String> store = builder.build();
+ final StateStore next = ((WrappedStateStore) store).wrapped();
+ assertInstanceOf(ChangeLoggingSessionBytesStoreWithHeaders.class, next);
+ }
+
+ @Test
+ public void shouldNotHaveChangeLoggingStoreWhenDisabled() {
+ setUp();
+ final SessionStoreWithHeaders<String, String> store = builder.withLoggingDisabled().build();
+ final StateStore next = ((WrappedStateStore) store).wrapped();
+ assertSame(sessionStoreWithHeaders, next);
+ }
+
+ @Test
+ public void shouldHaveCachingStoreWhenEnabled() {
+ setUp();
+ final SessionStoreWithHeaders<String, String> store = builder.withCachingEnabled().build();
+ final StateStore wrapped = ((WrappedStateStore) store).wrapped();
+ assertInstanceOf(MeteredSessionStoreWithHeaders.class, store);
+ assertInstanceOf(CachingSessionStore.class, wrapped);
+ }
+
+ @Test
+ public void shouldHaveChangeLoggingStoreWhenLoggingEnabled() {
+ setUp();
+ final SessionStoreWithHeaders<String, String> store = builder
+ .withLoggingEnabled(Collections.emptyMap())
+ .build();
+ final StateStore wrapped = ((WrappedStateStore) store).wrapped();
+ assertInstanceOf(MeteredSessionStoreWithHeaders.class, store);
+ assertInstanceOf(ChangeLoggingSessionBytesStoreWithHeaders.class, wrapped);
+ assertSame(sessionStoreWithHeaders, ((WrappedStateStore) wrapped).wrapped());
+ }
+
+ @Test
+ public void shouldHaveCachingAndChangeLoggingWhenBothEnabled() {
+ setUp();
+ final SessionStoreWithHeaders<String, String> store = builder
+ .withLoggingEnabled(Collections.emptyMap())
+ .withCachingEnabled()
+ .build();
+ final WrappedStateStore caching = (WrappedStateStore) ((WrappedStateStore) store).wrapped();
+ final WrappedStateStore changeLogging = (WrappedStateStore) caching.wrapped();
+ assertInstanceOf(MeteredSessionStoreWithHeaders.class, store);
+ assertInstanceOf(CachingSessionStore.class, caching);
+ assertInstanceOf(ChangeLoggingSessionBytesStoreWithHeaders.class, changeLogging);
+ assertSame(sessionStoreWithHeaders, changeLogging.wrapped());
+ }
+
+ @Test
+ public void shouldNotWrapHeadersByteStore() {
+ when(supplier.name()).thenReturn(STORE_NAME);
+ when(supplier.metricsScope()).thenReturn(METRICS_SCOPE);
+ when(supplier.get()).thenReturn(new RocksDBSessionStoreWithHeaders(
+ new SessionRocksDBSegmentedBytesStoreWithHeaders(
+ "name",
+ "metric-scope",
+ 10L,
+ 5L,
+ new SessionKeySchema())));
+
+ builder = new SessionStoreWithHeadersBuilder<>(
+ supplier,
+ Serdes.String(),
+ Serdes.String(),
+ new MockTime());
+
+ final SessionStoreWithHeaders<String, String> store = builder
+ .withLoggingDisabled()
+ .withCachingDisabled()
+ .build();
+ assertInstanceOf(RocksDBSessionStoreWithHeaders.class, ((WrappedStateStore) store).wrapped());
+ }
+
+ @Test
+ public void shouldWrapPlainStoreAsHeadersStore() {
+ when(supplier.name()).thenReturn(STORE_NAME);
+ when(supplier.metricsScope()).thenReturn(METRICS_SCOPE);
+ when(supplier.get()).thenReturn(new RocksDBSessionStore(
+ new RocksDBSegmentedBytesStore(
+ "name",
+ "metric-scope",
+ 10L,
+ 5L,
+ new SessionKeySchema())));
+
+ builder = new SessionStoreWithHeadersBuilder<>(
+ supplier,
+ Serdes.String(),
+ Serdes.String(),
+ new MockTime());
+
+ final SessionStoreWithHeaders<String, String> store = builder
+ .withLoggingDisabled()
+ .withCachingDisabled()
+ .build();
+ assertInstanceOf(SessionToHeadersStoreAdapter.class, ((WrappedStateStore) store).wrapped());
+ }
+
+ @Test
+ public void shouldThrowNullPointerIfInnerIsNull() {
+ assertThrows(NullPointerException.class, () -> new SessionStoreWithHeadersBuilder<>(null, Serdes.String(), Serdes.String(), new MockTime()));
+ }
+ }
+
+ /**
+ * End-to-end query behavior of the {@code withKey} form of
+ * {@link TimestampedWindowRangeWithHeadersQuery} against a real store built by the builder over
+ * each supported build path. Caching is disabled: the query reads the underlying store directly
+ * (it never consults the cache), so writes must be store-served.
+ */
+ @Nested
+ class QueryTests {
+ private static final String STORE_NAME = "test-session-store";
+ private static final String METRICS_SCOPE = "metrics-scope";
+ private static final long RETENTION = 60_000L;
+ private static final long SEGMENT_INTERVAL = 30_000L;
+ private static final long SESSION_START = 1_000L;
+ private static final long SESSION_END = 5_000L;
+
+ @Mock
+ private SessionBytesStoreSupplier supplier;
+
+ private SessionStore<Bytes, byte[]> innerStore(final StoreType storeType) {
+ switch (storeType) {
+ case NATIVE:
+ return new RocksDBSessionStoreWithHeaders(
+ new SessionRocksDBSegmentedBytesStoreWithHeaders(
+ STORE_NAME, METRICS_SCOPE, RETENTION, SEGMENT_INTERVAL, new SessionKeySchema()));
+ case ADAPTER:
+ return new RocksDBSessionStore(
+ new RocksDBSegmentedBytesStore(
+ STORE_NAME, METRICS_SCOPE, RETENTION, SEGMENT_INTERVAL, new SessionKeySchema()));
+ case IN_MEMORY:
+ // Non-persistent supplier: the builder wraps it with the in-memory headers marker,
+ // which (like the native store) persists the value-with-headers byte format.
+ return new InMemorySessionStore(STORE_NAME, RETENTION, METRICS_SCOPE);
+ default:
+ throw new IllegalArgumentException("unknown store type: " + storeType);
+ }
+ }
+
+ private SessionStoreWithHeaders<String, String> buildAndInitStore(final StoreType storeType) {
+ lenient().when(supplier.name()).thenReturn(STORE_NAME);
+ lenient().when(supplier.metricsScope()).thenReturn(METRICS_SCOPE);
+ lenient().when(supplier.segmentIntervalMs()).thenReturn(SEGMENT_INTERVAL);
+ lenient().when(supplier.retentionPeriod()).thenReturn(RETENTION);
+ lenient().when(supplier.get()).thenReturn(innerStore(storeType));
+
+ final SessionStoreWithHeaders<String, String> store =
+ new SessionStoreWithHeadersBuilder<>(supplier, Serdes.String(), Serdes.String(), new MockTime())
+ .withLoggingDisabled()
+ .withCachingDisabled()
+ .build();
+
+ final ThreadCache cache = new ThreadCache(new LogContext("test "), 0, new MockStreamsMetrics(new Metrics()));
+ final InternalMockProcessorContext<String, String> context = new InternalMockProcessorContext<>(
+ TestUtils.tempDirectory(), Serdes.String(), Serdes.String(), null, cache);
+ context.setRecordContext(new ProcessorRecordContext(0L, 0L, 0, "topic", new RecordHeaders()));
+ store.init(context, store);
+ return store;
+ }
+
+ private Headers headersWith(final String key, final String value) {
+ return new RecordHeaders().add(key, value.getBytes(StandardCharsets.UTF_8));
+ }
+
+ private QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> withKeyQuery(
+ final SessionStoreWithHeaders<String, String> store) {
+ return store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey("k"),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+ }
+
+ @ParameterizedTest
+ @CsvSource({"NATIVE", "IN_MEMORY"})
+ public void shouldReturnHeadersForTimestampedWindowRangeWithHeadersQueryOnHeaderPersistingStore(final StoreType storeType) {
+ // The native and in-memory builds both persist headers: the native store keeps them in its
+ // headers column family, and the in-memory marker stores the value bytes verbatim (the metered
+ // layer serializes the headers into those bytes). The adapter build is the one that drops them.
+ final SessionStoreWithHeaders<String, String> store = buildAndInitStore(storeType);
+ try {
+ final Headers headers = headersWith("h", "x");
+ store.put(new Windowed<>("k", new SessionWindow(SESSION_START, SESSION_END)), AggregationWithHeaders.make("v", headers));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = withKeyQuery(store);
+
+ assertTrue(result.isSuccess(), "Expected TimestampedWindowRangeWithHeadersQuery to succeed");
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals("k", record.key().key());
+ assertEquals(SESSION_START, record.key().window().start());
+ assertEquals(SESSION_END, record.key().window().end());
+ assertEquals("v", record.value());
+ // Timestamp is sourced from the session window's end (AggregationWithHeaders carries
+ // no timestamp of its own).
+ assertEquals(SESSION_END, record.timestamp());
+ assertEquals(headers, record.headers());
+ // The IQ result is a read-only snapshot: its headers are immutable (neither add nor remove).
+ assertThrows(IllegalStateException.class, () -> record.headers().add("new", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("h"));
+ assertFalse(iterator.hasNext());
+ }
+ assertNotNull(result.getPosition(), "Expected position to be set");
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldReturnEmptyHeadersForTimestampedWindowRangeWithHeadersQueryOnAdapterStore() {
+ // The adapter drops headers on write, so the record comes back with empty (never null)
+ // headers while value and timestamp (session window end) still round-trip.
+ final SessionStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.ADAPTER);
+ try {
+ store.put(new Windowed<>("k", new SessionWindow(SESSION_START, SESSION_END)), AggregationWithHeaders.make("v", headersWith("h", "x")));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = withKeyQuery(store);
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals("k", record.key().key());
+ assertEquals(SESSION_START, record.key().window().start());
+ assertEquals(SESSION_END, record.key().window().end());
+ assertEquals("v", record.value());
+ assertEquals(SESSION_END, record.timestamp());
+ assertEquals(new RecordHeaders(), record.headers());
+ assertThrows(IllegalStateException.class, () -> record.headers().add("new", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("h"));
+ assertFalse(iterator.hasNext());
+ }
+ assertNotNull(result.getPosition(), "Expected position to be set");
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldNotThrowForPlainSessionStoreSinceTimestampIsSessionWindowEnd() {
+ // Sessions have a single adapter path (SessionToHeadersStoreAdapter over a plain session store);
+ // there is no timestamped-vs-plain split like window stores have. Where a plain WINDOW store
+ // surfaces a stored ts=-1 and makes next() throw, a plain SESSION store never persists a
+ // per-record timestamp: the ReadOnlyRecord timestamp is the (always non-negative) session-window
+ // end, so iteration always succeeds.
+ final SessionStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.ADAPTER);
+ try {
+ store.put(new Windowed<>("k", new SessionWindow(SESSION_START, SESSION_END)),
+ AggregationWithHeaders.make("v", headersWith("h", "x")));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = withKeyQuery(store);
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ final ReadOnlyRecord<Windowed<String>, String> record = assertDoesNotThrow(iterator::next);
+ assertEquals(SESSION_END, record.timestamp());
+ assertFalse(iterator.hasNext());
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldRejectWithWindowStartRangeFormAgainstSessionStore() {
+ final SessionStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.NATIVE);
+ try {
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(0), Instant.ofEpochMilli(RETENTION)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+
+ assertFalse(result.isSuccess());
+ assertTrue(
+ result.getFailureMessage().contains("SessionStores only support TimestampedWindowRangeWithHeadersQuery.withKey"),
+ "unexpected message: " + result.getFailureMessage());
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldCollectExecutionInfoForTimestampedWindowRangeWithHeadersQueryWhenRequested() {
+ // With execution info enabled, the result must carry both the wrapped (native) store's entry
+ // and the metered handler's entry.
+ final SessionStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.NATIVE);
+ try {
+ store.put(new Windowed<>("k", new SessionWindow(SESSION_START, SESSION_END)), AggregationWithHeaders.make("v", headersWith("h", "x")));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withKey("k"),
+ PositionBound.unbounded(),
+ new QueryConfig(true));
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ final String info = String.join("\n", result.getExecutionInfo());
+ assertTrue(
+ info.contains(RocksDBSessionStoreWithHeaders.class.getName())
+ && info.contains(MeteredSessionStoreWithHeaders.class.getName()),
+ "execution info missing an entry: " + info);
+ // The metered handler reports the serdes it went through for a locally-handled query
+ // (only the metered layer emits "with serdes"; the native store's entry does not).
+ assertTrue(info.contains("with serdes"), "metered entry missing serdes detail: " + info);
+ // The metered entry's duration is measured with the injected (frozen MockTime) clock,
+ // so it reads a deterministic 0ns; the native store still times with System.nanoTime(),
+ // so only the metered entry can read " in 0ns". This pins timing to the injected clock.
+ assertTrue(info.contains(" in 0ns"), "metered timing not from injected clock: " + info);
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldNotCollectExecutionInfoForTimestampedWindowRangeWithHeadersQueryWhenNotRequested() {
+ final SessionStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.NATIVE);
+ try {
+ store.put(new Windowed<>("k", new SessionWindow(SESSION_START, SESSION_END)), AggregationWithHeaders.make("v", headersWith("h", "x")));
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = withKeyQuery(store);
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(result.getExecutionInfo().isEmpty(), "Expected no execution info: " + result.getExecutionInfo());
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldReturnIdenticalResultsForNativeAndAdapterBuiltStores() {
+ // Build-path parity for the existing (header-stripped) WindowRangeQuery.withKey: KIP-1356 makes
+ // the native store serve it exactly as the adapter build already did. withKey filters by key, so
+ // an interleaved "other" (noise) key must be excluded, and both builds must return only key k's
+ // sessions in session-start order.
+ // Session windows are variable-length (end is not start + a fixed size), so each session below
+ // has a different duration -- this verifies the session end round-trips from the raw result
+ // rather than being derived from any fixed window size.
+ final SessionStoreWithHeaders<String, String> nativeStore = buildAndInitStore(StoreType.NATIVE);
+ final SessionStoreWithHeaders<String, String> adapterStore = buildAndInitStore(StoreType.ADAPTER);
+ try {
+ // ascending session-start insertion into native, interleaved with a noise key
+ nativeStore.put(new Windowed<>("other", new SessionWindow(1_000L, 2_000L)), AggregationWithHeaders.make("o1", headersWith("h", "o")));
+ nativeStore.put(new Windowed<>("k", new SessionWindow(1_000L, 1_500L)), AggregationWithHeaders.make("v1", headersWith("h", "1")));
+ nativeStore.put(new Windowed<>("k", new SessionWindow(3_000L, 8_000L)), AggregationWithHeaders.make("v2", headersWith("h", "2")));
+ nativeStore.put(new Windowed<>("other", new SessionWindow(6_000L, 9_000L)), AggregationWithHeaders.make("o2", headersWith("h", "o")));
+ nativeStore.put(new Windowed<>("k", new SessionWindow(10_000L, 12_500L)), AggregationWithHeaders.make("v3", headersWith("h", "3")));
+ // descending session-start insertion into adapter (same sessions reversed), plus the noise key
+ adapterStore.put(new Windowed<>("k", new SessionWindow(10_000L, 12_500L)), AggregationWithHeaders.make("v3", headersWith("h", "3")));
+ adapterStore.put(new Windowed<>("other", new SessionWindow(6_000L, 9_000L)), AggregationWithHeaders.make("o2", headersWith("h", "o")));
+ adapterStore.put(new Windowed<>("k", new SessionWindow(3_000L, 8_000L)), AggregationWithHeaders.make("v2", headersWith("h", "2")));
+ adapterStore.put(new Windowed<>("k", new SessionWindow(1_000L, 1_500L)), AggregationWithHeaders.make("v1", headersWith("h", "1")));
+ adapterStore.put(new Windowed<>("other", new SessionWindow(1_000L, 2_000L)), AggregationWithHeaders.make("o1", headersWith("h", "o")));
+
+ // Each "k" session has a distinct length (500, 5000, 2500 ms); "other" is filtered out.
+ final List<KeyValue<Windowed<String>, String>> expected = List.of(
+ KeyValue.pair(new Windowed<>("k", new SessionWindow(1_000L, 1_500L)), "v1"),
+ KeyValue.pair(new Windowed<>("k", new SessionWindow(3_000L, 8_000L)), "v2"),
+ KeyValue.pair(new Windowed<>("k", new SessionWindow(10_000L, 12_500L)), "v3"));
+
+ assertEquals(expected, plainWindowRangeResults(nativeStore),
+ "native WindowRangeQuery.withKey should return only key k's sessions in session-start order");
+ assertEquals(expected, plainWindowRangeResults(adapterStore),
+ "adapter WindowRangeQuery.withKey should return only key k's sessions in session-start order");
+ } finally {
+ nativeStore.close();
+ adapterStore.close();
+ }
+ }
+
+ // Drains the (header-stripped) plain WindowRangeQuery.withKey, keeping the full windowed key and
+ // value -- used to compare native and adapter build paths.
+ private List<KeyValue<Windowed<String>, String>> plainWindowRangeResults(final SessionStoreWithHeaders<String, String> store) {
+ final WindowRangeQuery<String, String> query = WindowRangeQuery.withKey("k");
+ final List<KeyValue<Windowed<String>, String>> out = new ArrayList<>();
+ try (KeyValueIterator<Windowed<String>, String> iterator =
+ store.query(query, PositionBound.unbounded(), new QueryConfig(false)).getResult()) {
+ while (iterator.hasNext()) {
+ out.add(iterator.next());
+ }
+ }
+ return out;
+ }
+ }
+}
diff --git a/streams/src/test/java/org/apache/kafka/streams/state/internals/TimestampedWindowStoreWithHeadersBuilderTest.java b/streams/src/test/java/org/apache/kafka/streams/state/internals/TimestampedWindowStoreWithHeadersBuilderTest.java
index 093d5c8022f..1be480df804 100644
--- a/streams/src/test/java/org/apache/kafka/streams/state/internals/TimestampedWindowStoreWithHeadersBuilderTest.java
+++ b/streams/src/test/java/org/apache/kafka/streams/state/internals/TimestampedWindowStoreWithHeadersBuilderTest.java
@@ -27,6 +27,7 @@ import org.apache.kafka.common.utils.internals.LogContext;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.errors.StreamsException;
import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.TimeWindow;
import org.apache.kafka.streams.processor.StateStore;
import org.apache.kafka.streams.processor.api.ReadOnlyRecord;
import org.apache.kafka.streams.processor.internals.MockStreamsMetrics;
@@ -35,7 +36,10 @@ import org.apache.kafka.streams.query.PositionBound;
import org.apache.kafka.streams.query.QueryConfig;
import org.apache.kafka.streams.query.QueryResult;
import org.apache.kafka.streams.query.TimestampedWindowKeyWithHeadersQuery;
+import org.apache.kafka.streams.query.TimestampedWindowRangeWithHeadersQuery;
import org.apache.kafka.streams.query.WindowKeyQuery;
+import org.apache.kafka.streams.query.WindowRangeQuery;
+import org.apache.kafka.streams.state.KeyValueIterator;
import org.apache.kafka.streams.state.ReadOnlyRecordIterator;
import org.apache.kafka.streams.state.TimestampedWindowStoreWithHeaders;
import org.apache.kafka.streams.state.ValueAndTimestamp;
@@ -245,9 +249,10 @@ public class TimestampedWindowStoreWithHeadersBuilderTest {
}
/**
- * End-to-end query behavior of {@link TimestampedWindowKeyWithHeadersQuery} against a real store
- * built by the builder over each supported build path. Caching is disabled: a window key query reads
- * the underlying store directly (it never consults the cache), so writes must be store-served.
+ * End-to-end query behavior of {@link TimestampedWindowKeyWithHeadersQuery} and the window-store
+ * ({@code withWindowStartRange}) form of {@link TimestampedWindowRangeWithHeadersQuery} against a real
+ * store built by the builder over each supported build path. Caching is disabled: these queries read
+ * the underlying store directly (they never consult the cache), so writes must be store-served.
*/
@Nested
class QueryTests {
@@ -320,6 +325,15 @@ public class TimestampedWindowStoreWithHeadersBuilderTest {
new QueryConfig(false));
}
+ private QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> windowRangeQuery(
+ final TimestampedWindowStoreWithHeaders<String, String> store) {
+ return store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(0), Instant.ofEpochMilli(RETENTION)),
+ PositionBound.unbounded(),
+ new QueryConfig(false));
+ }
+
@ParameterizedTest
@CsvSource({"NATIVE", "IN_MEMORY"})
public void shouldReturnHeadersForTimestampedWindowKeyWithHeadersQueryOnHeaderPersistingStore(final StoreType storeType) {
@@ -449,6 +463,13 @@ public class TimestampedWindowStoreWithHeadersBuilderTest {
info.contains(RocksDBTimestampedWindowStoreWithHeaders.class.getName())
&& info.contains(MeteredTimestampedWindowStoreWithHeaders.class.getName()),
"execution info missing an entry: " + info);
+ // The metered handler reports the serdes it went through for a locally-handled query
+ // (only the metered layer emits "with serdes"; the native store's entry does not).
+ assertTrue(info.contains("with serdes"), "metered entry missing serdes detail: " + info);
+ // The metered entry's duration is measured with the injected (frozen MockTime) clock,
+ // so it reads a deterministic 0ns; the native store still times with System.nanoTime(),
+ // so only the metered entry can read " in 0ns". This pins timing to the injected clock.
+ assertTrue(info.contains(" in 0ns"), "metered timing not from injected clock: " + info);
}
} finally {
store.close();
@@ -472,8 +493,166 @@ public class TimestampedWindowStoreWithHeadersBuilderTest {
}
}
+ @ParameterizedTest
+ @CsvSource({"NATIVE", "IN_MEMORY"})
+ public void shouldReturnHeadersForTimestampedWindowRangeWithHeadersQueryOnHeaderPersistingStore(final StoreType storeType) {
+ // Mirrors the window-key case: the native and in-memory builds persist headers (native in its
+ // headers column family, in-memory via the value-with-headers marker); the adapter build drops them.
+ final TimestampedWindowStoreWithHeaders<String, String> store = buildAndInitStore(storeType);
+ try {
+ final Headers headers = headersWith("h", "x");
+ store.put("k", ValueTimestampHeaders.make("v", 1_005L, headers), WINDOW_START);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = windowRangeQuery(store);
+
+ assertTrue(result.isSuccess(), "Expected TimestampedWindowRangeWithHeadersQuery to succeed");
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals("k", record.key().key());
+ assertEquals(WINDOW_START, record.key().window().start());
+ assertEquals(WINDOW_START + WINDOW_SIZE, record.key().window().end());
+ assertEquals("v", record.value());
+ assertEquals(1_005L, record.timestamp());
+ assertEquals(headers, record.headers());
+ // The IQ result is a read-only snapshot: its headers are immutable (neither add nor remove).
+ assertThrows(IllegalStateException.class, () -> record.headers().add("new", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("h"));
+ assertFalse(iterator.hasNext());
+ }
+ assertNotNull(result.getPosition(), "Expected position to be set");
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldReturnEmptyHeadersForTimestampedWindowRangeWithHeadersQueryOnAdapterStore() {
+ // The timestamped adapter keeps the timestamp but drops headers on write, so the record comes
+ // back with empty (never null) headers while value and timestamp still round-trip.
+ final TimestampedWindowStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.ADAPTER);
+ try {
+ store.put("k", ValueTimestampHeaders.make("v", 1_005L, headersWith("h", "x")), WINDOW_START);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = windowRangeQuery(store);
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(iterator.hasNext());
+ final ReadOnlyRecord<Windowed<String>, String> record = iterator.next();
+ assertEquals("k", record.key().key());
+ assertEquals(WINDOW_START, record.key().window().start());
+ assertEquals("v", record.value());
+ assertEquals(1_005L, record.timestamp());
+ assertEquals(new RecordHeaders(), record.headers());
+ // Even the empty headers are a read-only snapshot: neither add nor remove is allowed.
+ assertThrows(IllegalStateException.class, () -> record.headers().add("new", new byte[0]));
+ assertThrows(IllegalStateException.class, () -> record.headers().remove("h"));
+ assertFalse(iterator.hasNext());
+ }
+ assertNotNull(result.getPosition(), "Expected position to be set");
+ } finally {
+ store.close();
+ }
+ }
+
@Test
- public void shouldReturnIdenticalResultsForNativeAndAdapterBuiltStores() {
+ public void shouldThrowForTimestampedWindowRangeWithHeadersQueryOnPlainSupplier() {
+ // A plain (non-timestamped) window supplier surfaces every entry with timestamp = -1, which
+ // cannot be represented as a ReadOnlyRecord. The query succeeds, but iterating throws.
+ final TimestampedWindowStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.PLAIN_ADAPTER);
+ try {
+ store.put("k", ValueTimestampHeaders.make("v", 1_005L, headersWith("h", "x")), WINDOW_START);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = windowRangeQuery(store);
+
+ assertTrue(result.isSuccess(), "The query itself succeeds; the failure surfaces while iterating");
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ final StreamsException e = assertThrows(StreamsException.class, iterator::next,
+ "An entry with ts=-1 cannot be represented as a ReadOnlyRecord");
+ assertTrue(e.getMessage().contains("as a ReadOnlyRecord") && e.getMessage().contains("is negative"),
+ "unexpected message: " + e.getMessage());
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldThrowForNegativeStoredTimestampForTimestampedWindowRangeWithHeadersQuery() {
+ // A caller can store a negative timestamp directly on a native (header-persisting) store.
+ // The query succeeds, but the lazily-evaluated iterator throws while advancing (a ReadOnlyRecord
+ // timestamp must be non-negative).
+ final TimestampedWindowStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.NATIVE);
+ try {
+ store.put("k", ValueTimestampHeaders.make("v", -1L, headersWith("h", "x")), WINDOW_START);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = windowRangeQuery(store);
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ final StreamsException e = assertThrows(StreamsException.class, iterator::next);
+ assertTrue(e.getMessage().contains("as a ReadOnlyRecord") && e.getMessage().contains("is negative"),
+ "unexpected message: " + e.getMessage());
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldCollectExecutionInfoForTimestampedWindowRangeWithHeadersQueryWhenRequested() {
+ // With execution info enabled, the result must carry both the wrapped (native) store's entry
+ // and the metered handler's entry.
+ final TimestampedWindowStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.NATIVE);
+ try {
+ store.put("k", ValueTimestampHeaders.make("v", 1_005L, headersWith("h", "x")), WINDOW_START);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = store.query(
+ TimestampedWindowRangeWithHeadersQuery.<String, String>withWindowStartRange(
+ Instant.ofEpochMilli(0), Instant.ofEpochMilli(RETENTION)),
+ PositionBound.unbounded(),
+ new QueryConfig(true));
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ final String info = String.join("\n", result.getExecutionInfo());
+ assertTrue(
+ info.contains(RocksDBTimestampedWindowStoreWithHeaders.class.getName())
+ && info.contains(MeteredTimestampedWindowStoreWithHeaders.class.getName()),
+ "execution info missing an entry: " + info);
+ // The metered handler reports the serdes it went through for a locally-handled query
+ // (only the metered layer emits "with serdes"; the native store's entry does not).
+ assertTrue(info.contains("with serdes"), "metered entry missing serdes detail: " + info);
+ // The metered entry's duration is measured with the injected (frozen MockTime) clock,
+ // so it reads a deterministic 0ns; the native store still times with System.nanoTime(),
+ // so only the metered entry can read " in 0ns". This pins timing to the injected clock.
+ assertTrue(info.contains(" in 0ns"), "metered timing not from injected clock: " + info);
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldNotCollectExecutionInfoForTimestampedWindowRangeWithHeadersQueryWhenNotRequested() {
+ final TimestampedWindowStoreWithHeaders<String, String> store = buildAndInitStore(StoreType.NATIVE);
+ try {
+ store.put("k", ValueTimestampHeaders.make("v", 1_005L, headersWith("h", "x")), WINDOW_START);
+
+ final QueryResult<ReadOnlyRecordIterator<Windowed<String>, String>> result = windowRangeQuery(store);
+
+ assertTrue(result.isSuccess());
+ try (ReadOnlyRecordIterator<Windowed<String>, String> iterator = result.getResult()) {
+ assertTrue(result.getExecutionInfo().isEmpty(), "Expected no execution info: " + result.getExecutionInfo());
+ }
+ } finally {
+ store.close();
+ }
+ }
+
+ @Test
+ public void shouldReturnIdenticalResultsForNativeAndAdapterBuiltStoresForWindowKeyQuery() {
// Build-path parity for the existing (header-stripped) WindowKeyQuery: KIP-1356 makes the native
// store serve it exactly as the adapter build already did.
final ValueTimestampHeaders<String> v1 = ValueTimestampHeaders.make("v1", 1_005L, headersWith("h", "1"));
@@ -513,6 +692,53 @@ public class TimestampedWindowStoreWithHeadersBuilderTest {
}
}
+ @Test
+ public void shouldReturnIdenticalResultsForNativeAndAdapterBuiltStoresForWindowRangeQuery() {
+ // Build-path parity for the existing (header-stripped) WindowRangeQuery.withWindowStartRange:
+ // KIP-1356 makes the native store serve it exactly as the adapter build already did. Unlike
+ // WindowKeyQuery, withWindowStartRange has no key filter, so an interleaved "other" (noise) key
+ // is returned too -- both builds must yield every key, ordered by window-start then key.
+ final ValueTimestampHeaders<String> v1 = ValueTimestampHeaders.make("v1", 1_005L, headersWith("h", "1"));
+ final ValueTimestampHeaders<String> v2 = ValueTimestampHeaders.make("v2", 3_005L, headersWith("h", "2"));
+ final ValueTimestampHeaders<String> v3 = ValueTimestampHeaders.make("v3", 5_005L, headersWith("h", "3"));
+ final ValueTimestampHeaders<String> o1 = ValueTimestampHeaders.make("o1", 2_005L, headersWith("h", "o"));
+ final ValueTimestampHeaders<String> o2 = ValueTimestampHeaders.make("o2", 4_005L, headersWith("h", "o"));
+
+ final TimestampedWindowStoreWithHeaders<String, String> nativeStore = buildAndInitStore(StoreType.NATIVE);
+ final TimestampedWindowStoreWithHeaders<String, String> adapterStore = buildAndInitStore(StoreType.ADAPTER);
+ try {
+ // ascending window-start insertion order, interleaved with a noise key at shared windows
+ nativeStore.put("other", o1, WINDOW_START);
+ nativeStore.put("k", v1, WINDOW_START);
+ nativeStore.put("k", v2, WINDOW_START + 2_000L);
+ nativeStore.put("other", o2, WINDOW_START + 2_000L);
+ nativeStore.put("k", v3, WINDOW_START + 4_000L);
+ // descending window-start insertion order (same windows reversed), plus the noise key
+ adapterStore.put("k", v3, WINDOW_START + 4_000L);
+ adapterStore.put("other", o2, WINDOW_START + 2_000L);
+ adapterStore.put("k", v2, WINDOW_START + 2_000L);
+ adapterStore.put("k", v1, WINDOW_START);
+ adapterStore.put("other", o1, WINDOW_START);
+
+ // withWindowStartRange returns every key, ordered by key then window-start (all of "k"
+ // ascending, then all of "other" ascending) -- identical across both build paths.
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<String>>> expected = List.of(
+ KeyValue.pair(windowedKey("k", WINDOW_START), ValueAndTimestamp.make("v1", 1_005L)),
+ KeyValue.pair(windowedKey("k", WINDOW_START + 2_000L), ValueAndTimestamp.make("v2", 3_005L)),
+ KeyValue.pair(windowedKey("k", WINDOW_START + 4_000L), ValueAndTimestamp.make("v3", 5_005L)),
+ KeyValue.pair(windowedKey("other", WINDOW_START), ValueAndTimestamp.make("o1", 2_005L)),
+ KeyValue.pair(windowedKey("other", WINDOW_START + 2_000L), ValueAndTimestamp.make("o2", 4_005L)));
+
+ assertEquals(expected, plainWindowRangeResults(nativeStore),
+ "native WindowRangeQuery should return every key in (key, window-start) order");
+ assertEquals(expected, plainWindowRangeResults(adapterStore),
+ "adapter WindowRangeQuery should return every key in (key, window-start) order");
+ } finally {
+ nativeStore.close();
+ adapterStore.close();
+ }
+ }
+
// Drains the (header-stripped) plain WindowKeyQuery, which yields the window-start timestamp keyed
// to a ValueAndTimestamp -- used to compare native and adapter build paths.
private List<KeyValue<Long, ValueAndTimestamp<String>>> plainWindowKeyResults(
@@ -528,5 +754,25 @@ public class TimestampedWindowStoreWithHeadersBuilderTest {
}
return out;
}
+
+ private Windowed<String> windowedKey(final String key, final long windowStart) {
+ return new Windowed<>(key, new TimeWindow(windowStart, windowStart + WINDOW_SIZE));
+ }
+
+ // Drains the (header-stripped) plain WindowRangeQuery.withWindowStartRange, keeping the full
+ // windowed key and ValueAndTimestamp -- used to compare native and adapter build paths.
+ private List<KeyValue<Windowed<String>, ValueAndTimestamp<String>>> plainWindowRangeResults(
+ final TimestampedWindowStoreWithHeaders<String, String> store) {
+ final WindowRangeQuery<String, ValueAndTimestamp<String>> query =
+ WindowRangeQuery.withWindowStartRange(Instant.ofEpochMilli(0), Instant.ofEpochMilli(RETENTION));
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<String>>> out = new ArrayList<>();
+ try (KeyValueIterator<Windowed<String>, ValueAndTimestamp<String>> iterator =
+ store.query(query, PositionBound.unbounded(), new QueryConfig(false)).getResult()) {
+ while (iterator.hasNext()) {
+ out.add(iterator.next());
+ }
+ }
+ return out;
+ }
}
}