You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/07/17 18:34:50 UTC
[GitHub] [kafka] lct45 opened a new pull request #9039: KIP-450 SlidingWindows [POC]
lct45 opened a new pull request #9039:
URL: https://github.com/apache/kafka/pull/9039
Updating basic classes for SlidingWindows implementation, WIP and POC, things will shift as KIP discussion continues.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463088582
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
Review comment:
changed to
` if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
} else {
//left window just contains the current record
valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465339322
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
Review comment:
fixed for both
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r464728426
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/TimeWindowedKStreamImpl.java
##########
@@ -19,7 +19,16 @@
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
-import org.apache.kafka.streams.kstream.*;
Review comment:
The build should actually fail on wildcard imports... Do we have some checkstyle gaps? @lct45 can you maybe look into that (if not, also ok).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465079294
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
Review comment:
We want to be able to find the furthest window for which we can create a corresponding right window, so for any record the furthest window we will ever need will start at `timestamp - 2 * timeDifference`, but we will need to have these around to calculate new windows, hence the longer retention time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463076565
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
Review comment:
updated above! changed to `prevRightWindowPossible` , lmk if that still seems confusing. I definitely kept getting them all mixed up so I think this will help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r467260318
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
Review comment:
Sounds good, I'll change that when I implement the reverse iterator in the next PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r474977836
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,579 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+ builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(MockInitializer.STRING_INIT, MockAggregator.toStringInstance("+"), Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()));
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput(null, "1");
+ assertThat(appender.getMessages(), hasItem("Skipping record due to null key or value. value=[1] topic=[topic] partition=[0] offset=[0]"));
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingExpiredWindowByGrace() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KStream<String, String> stream1 = builder.stream(topic, Consumed.with(Serdes.String(), Serdes.String()));
+ stream1.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(90L)))
+ .aggregate(
+ () -> "",
+ MockAggregator.toStringInstance("+"),
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()).withCachingDisabled().withLoggingDisabled()
+ )
+ .toStream()
+ .map((key, value) -> new KeyValue<>(key.toString(), value))
+ .to("output");
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("k", "100", 200L);
+ inputTopic.pipeInput("k", "0", 100L);
+ inputTopic.pipeInput("k", "1", 101L);
+ inputTopic.pipeInput("k", "2", 102L);
+ inputTopic.pipeInput("k", "3", 103L);
+ inputTopic.pipeInput("k", "4", 104L);
+ inputTopic.pipeInput("k", "5", 105L);
+ inputTopic.pipeInput("k", "6", 15L);
+
+ assertLatenessMetrics(driver, is(7.0), is(185.0), is(96.25));
+
+ assertThat(appender.getMessages(), hasItems(
+ // left window for k@100
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[1] timestamp=[100] window=[90,100] expiration=[110] streamTime=[200]",
+ // left window for k@101
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[2] timestamp=[101] window=[91,101] expiration=[110] streamTime=[200]",
+ // left window for k@102
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[3] timestamp=[102] window=[92,102] expiration=[110] streamTime=[200]",
+ // left window for k@103
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[4] timestamp=[103] window=[93,103] expiration=[110] streamTime=[200]",
+ // left window for k@104
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[5] timestamp=[104] window=[94,104] expiration=[110] streamTime=[200]",
+ // left window for k@105
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[6] timestamp=[105] window=[95,105] expiration=[110] streamTime=[200]",
+ // left window for k@15
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[7] timestamp=[15] window=[5,15] expiration=[110] streamTime=[200]"
+ ));
+ final TestOutputTopic<String, String> outputTopic =
+ driver.createOutputTopic("output", new StringDeserializer(), new StringDeserializer());
+ assertThat(outputTopic.readRecord(), equalTo(new TestRecord<>("[k@190/200]", "+100", null, 200L)));
+ assertTrue(outputTopic.isEmpty());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testProcessorRandomInput() {
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(10000)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+
+ final List<ValueAndTimestamp<String>> input = Arrays.asList(
+ ValueAndTimestamp.make("A", 10L),
+ ValueAndTimestamp.make("A", 15L),
+ ValueAndTimestamp.make("A", 16L),
+ ValueAndTimestamp.make("A", 18L),
+ ValueAndTimestamp.make("A", 30L),
+ ValueAndTimestamp.make("A", 40L),
+ ValueAndTimestamp.make("A", 55L),
+ ValueAndTimestamp.make("A", 56L),
+ ValueAndTimestamp.make("A", 58L),
+ ValueAndTimestamp.make("A", 58L),
+ ValueAndTimestamp.make("A", 62L),
+ ValueAndTimestamp.make("A", 63L),
+ ValueAndTimestamp.make("A", 63L),
+ ValueAndTimestamp.make("A", 63L),
+ ValueAndTimestamp.make("A", 76L),
+ ValueAndTimestamp.make("A", 77L),
+ ValueAndTimestamp.make("A", 80L)
+ );
+
+ final long seed = new Random().nextLong();
+ final Random shuffle = new Random(seed);
+ Collections.shuffle(input, shuffle);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ for (int i = 0; i < input.size(); i++) {
+ inputTopic1.pipeInput("A", input.get(i).value(), input.get(i).timestamp());
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> results = new HashMap<>();
+
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (results.putIfAbsent(start, valueAndTimestamp) != null) {
+ results.replace(start, valueAndTimestamp);
+ }
+ }
+ randomEqualityCheck(results, seed);
+ }
+
+ private void randomEqualityCheck(final Map<Long, ValueAndTimestamp<String>> actual, final Long seed) {
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+A", 10L));
Review comment:
Just a minor note, can we order the expected results by window timestamp?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468218540
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedCogroupedKStreamImpl.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedCogroupedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+public class SlidingWindowedCogroupedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedCogroupedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+ private final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns;
+
+ SlidingWindowedCogroupedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode,
+ final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns) {
+ super(name, null, null, subTopologySourceNodes, streamsGraphNode, builder);
+ //keySerde and valueSerde are null because there are many different groupStreams that they could be from
+ this.windows = windows;
+ this.aggregateBuilder = aggregateBuilder;
+ this.groupPatterns = groupPatterns;
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer) {
+ return aggregate(initializer, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named) {
+ return aggregate(initializer, named, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal = new MaterializedInternal<>(
+ materialized,
+ builder,
+ CogroupedKStreamImpl.AGGREGATE_NAME);
+ return aggregateBuilder.build(
+ groupPatterns,
+ initializer,
+ new NamedInternal(named),
+ materialize(materializedInternal),
+ materializedInternal.keySerde() != null ?
+ new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs())
+ : null,
+ materializedInternal.valueSerde(),
+ materializedInternal.queryableStoreName(),
+ null,
+ windows,
+ null,
+ null);
+ }
+
+ private StoreBuilder<TimestampedWindowStore<K, V>> materialize(
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materialized) {
Review comment:
I think we usually leave the arguments on the same line as the method declaration (even if that line ends up way too long)
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KGroupedStreamImplTest.java
##########
@@ -77,82 +80,212 @@ public void before() {
groupedStream = stream.groupByKey(Grouped.with(Serdes.String(), Serdes.String()));
}
- @Test(expected = NullPointerException.class)
Review comment:
Awesome, thanks for cleaning up some of these older tests 😄
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
Review comment:
Comment on the reverse case left behind
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+
Review comment:
nit: extra line breaks
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedCogroupedKStreamImpl.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedCogroupedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+public class SlidingWindowedCogroupedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedCogroupedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+ private final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns;
+
+ SlidingWindowedCogroupedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode,
+ final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns) {
+ super(name, null, null, subTopologySourceNodes, streamsGraphNode, builder);
+ //keySerde and valueSerde are null because there are many different groupStreams that they could be from
+ this.windows = windows;
+ this.aggregateBuilder = aggregateBuilder;
+ this.groupPatterns = groupPatterns;
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer) {
+ return aggregate(initializer, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named) {
+ return aggregate(initializer, named, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal = new MaterializedInternal<>(
+ materialized,
+ builder,
+ CogroupedKStreamImpl.AGGREGATE_NAME);
+ return aggregateBuilder.build(
+ groupPatterns,
+ initializer,
+ new NamedInternal(named),
+ materialize(materializedInternal),
+ materializedInternal.keySerde() != null ?
+ new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs())
+ : null,
+ materializedInternal.valueSerde(),
+ materializedInternal.queryableStoreName(),
+ null,
+ windows,
+ null,
+ null);
+ }
+
+ private StoreBuilder<TimestampedWindowStore<K, V>> materialize(
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
Review comment:
Kind of hard to tell, but is the alignment in this method a bit off? Might be good to just highlight and auto-indent everything, intellij will take care of any issues if it's configured properly
##########
File path: checkstyle/suppressions.xml
##########
@@ -163,6 +163,12 @@
<suppress checks="(FinalLocalVariable|UnnecessaryParentheses|BooleanExpressionComplexity|CyclomaticComplexity|WhitespaceAfter|LocalVariableName)"
files="Murmur3.java"/>
+ <suppress checks="(NPathComplexity|MethodLength|CyclomaticComplexity)"
+ files="KStreamSlidingWindowAggregate.java"/>
+
+ <suppress checks="(CyclomaticComplexity)"
+ files="CogroupedStreamAggregateBuilder.java"/>
+
Review comment:
You should check to make sure all of these are still needed. In particular I bet we can get rid of the CogroupedStreamAggregateBuilder suppression once your cleanup PR is merged and this one is rebased
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
Review comment:
The input is the same for each test so the output is too, right? Maybe we can we pull all the output verification into a single method
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
Review comment:
nit: extra spaces after the `->`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding windows are defined by a record's timestamp, with window size based on the given maximum time difference (inclusive) between
+ * records in the same window and a given window grace period. While the window is sliding over the input data stream, a new window is
+ * created each time a record enters the sliding window or a record drops out of the sliding window.
+ * <p>
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * <ul>
+ * <li>window {@code [3000;8000]} contains [1] (created when first record enters the window)</li>
+ * <li>window {@code [4200;9200]} contains [1,2] (created when second record enters the window)</li>
+ * <li>window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)</li>
+ * <li>window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)</li>
+ * <li>window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)</li>
+ * </ul>
+ *<p>
+ * Note that while SlidingWindows are of a fixed size, as are {@link TimeWindows}, the start and end points of the window
+ * depend on when events occur in the stream (i.e., event timestamps), similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(SlidingWindows)
+ * @see CogroupedKStream#windowedBy(SlidingWindows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period. Reject out-of-order events that arrive after {@code grace}.
+ * A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size is < 0 or grace <= 0, or either can't be represented as {@code long milliseconds}
Review comment:
This comment needs to be updated, looks like we do allow a grace period of zero in the code/tests
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/CogroupedKStreamImplTest.java
##########
@@ -145,6 +146,11 @@ public void shouldNotHaveNullWindowOnWindowedBySession() {
cogroupedStream.windowedBy((SessionWindows) null);
}
+ @Test(expected = NullPointerException.class)
Review comment:
`assertThrows` 🙂
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
Review comment:
https://github.com/apache/kafka/pull/9157
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
Review comment:
Can we add some tests to verify the other Materialized properties, specifically the retention? You can just pick a single operator (eg `reduce`) and write a test to make sure data is available (only) within the retention period.
Also, do you think we can write a test to verify that the default retention is as expected when we don't specify it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466050097
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
Review comment:
Well, if the first window isn't the right window then we just wouldn't call `iterator.next` again, right? So we wouldn't have to do anything at all
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465379071
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
Not saying all this needs to be cleaned up in this PR. If we check one thing (eg `sessionMerger`) then we should check everything (eg `sessionMerged != null && sessionWindows != null`). We can decide whether we really need to check anything as followup
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466618450
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
I do think we'd need separate `build` methods, since that's where we originally accept multiple windows as arguments (where all but one type is set to null in each caller). But most of `build` doesn't touch the windows arguments so you could probably factor out all the window-independent code into a single method and just have each `build` method call that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463092579
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
Review comment:
Changing both of the `if()` for creating new windows cut down like half of the booleans too, which I think is good
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463083821
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
Review comment:
yes!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468814368
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
Review comment:
Yeah sorry I should have been more clear, I just meant push some data through and try to query the store to make sure it is/isn't there according to the retention period. You're right, it's not directly exposed anywhere
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463026843
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
The original just had the check for `sessionMerger != null`, are there scenarios where the sessionMerger would be null but the sessionWindows wouldn't? I did think it was kind of inconsistent to check 'sessionMerger' just that one time and check 'sessionWindows' the other times so maybe it was a mistake
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478715664
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
Review comment:
No, the condition is correct. In this context "early" just means "within timeDifferenceMs of the zero timestamp". We need some special handling to cover this full range of all record timestamps due to the inability to store negative timestamps. This algorithm works correctly for all records outside of this regardless of "now"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465751597
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
Review comment:
Hmmmm yeah, I think if there's something in the left window then we will always initialize `leftWinAgg` to something other than null, good catch. We essentially check `leftWinAgg.timestamp() < timestamp` in the while loop, so I don't think that should cause a problem, and `leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()` will never be true because that window would be out of range, right? And we only take the first left agg. Long way of saying, I think we can do the null check and nothing else but will know slightly more when we can test it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465073439
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
Would checking both be redundant? It looks like the method that ultimately calls this one will check that sessionMerger is not null for session windows, so I think either both of these will be null or neither will be null
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r464731092
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
Review comment:
`are processed` -> sounds like processing time semantics; maybe better `occur in the stream (i.e., event timestamps)`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
Review comment:
nit: double `/**`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
Review comment:
`and [a] given` ?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs <= 0) {
+ throw new IllegalArgumentException("Window timeDifference (timeDifference) must be larger than zero.");
+ }
+ final String msgPrefixGrace = prepareMillisCheckFailMsgPrefix(grace, "afterWindowEnd");
+ final long graceMs = ApiUtils.validateMillisecondDuration(grace, msgPrefixGrace);
+ if (graceMs < 0) {
+ throw new IllegalArgumentException("Grace period must not be negative.");
Review comment:
For consistency: `Grace period (grace) must` ? Frankly, I am not sure if we need to have "natural language" and the parameter name in those error messages -- also above. But we should do it in a consistent manner IMHO.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
Review comment:
as above (won't comment on this again) -- please address throughput the whole PR.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs <= 0) {
+ throw new IllegalArgumentException("Window timeDifference (timeDifference) must be larger than zero.");
Review comment:
`timeDifference (timeDifference)` (redundant) should be `time difference (timeDifference)`
`must be larger than zero.` -> `must not be negative.`
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
Review comment:
This line seems to be unnecessary for this test?
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForNegativeCases() {
+
+ verifyInEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(2)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyInEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(9)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+
+ verifyInEquality(
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(4), ofMillis(2)),
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(2))
+ );
+
+ assertNotEquals(
Review comment:
What is the difference between `verifyInEquality` and `assertNotEquals` ?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
Review comment:
as above.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
Review comment:
nit: if you want to have a new paragraph, you need to insert `<p>` tag -- otherwise, the empty line is just ignored all it's going to be one paragraph. -- If you don't want a paragraph, please remove the empty line.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
Review comment:
type `[w]indows`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
Review comment:
We must use HTML list markup to get bullet points rendered, ie, `<ul>` and `<li>` (cf https://www.w3schools.com/html/html_lists.asp)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
Review comment:
Reference to `CogroupedKStream` is missing
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
Review comment:
Beside the fact, that zero should be valid IMHO, what do we gain by testing `0` and `-1` ?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
Review comment:
I think we should also drop if `value == null` ? (It seem this `null` check is missing in the existing time/session-window aggregate processors, too)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
Review comment:
I guess a `timeDifference` of zero should be allowed to define a sliding window of size `1ms`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs <= 0) {
Review comment:
as above -> should be `timeDifferenceMs < 0`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
+ Instant.ofEpochMilli(timestamp + 1))
Review comment:
Why `+ 1` ?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
Review comment:
No need to pass in an `Instant` -- we should just pass in the `long` directly.
It might not be clear from the type hierarchy, but the overloads that accept `long` are only deprecated for the `ReadOnlyXxx` store, but are still available on the "read/write" stores to avoid unnecessary runtime overhead.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
Review comment:
Maybe we have the same issue in other processors, too (we might even have a ticket for it?) but won't we need to preserve `observedStreamTime` across restarts? It's transient atm... (Just want to confirm -- maybe it's ok as other processor do it the same way and we need to fix if for all of them at once?)
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
Maybe add comment to clarify which input should trigger which output:
```
// process A @ 2000ms
new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
// process A @ 2500ms
new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchRightWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(secondBatchLeftWindow, Long.MAX_VALUE)), "A:A", secondBatchTimestamp),
// process A @ 2900ms
...
```
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
Review comment:
nit: move `key` to the next line
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
Review comment:
I think we should do this check in `init` and use a `Runnable` that we just call blindly (ie, depending on the check, we instantiate the one or other `Runnable` and each `Runnable` implements a different algorithm.
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
Review comment:
This is also a pattern we try to move off. Use `assertThrows` instead.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
Review comment:
The comment seems redundant -- it just says exactly what the next line of code says.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
+
+ if ((windows.timeDifferenceMs() * 2 + windows.gracePeriodMs()) > retentionPeriod) {
Review comment:
why `* 2` ?
##########
File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStreamThreadTest.java
##########
@@ -221,7 +221,6 @@ public void shouldTransitionToRunningOnStart() throws Exception {
globalStreamThread.shutdown();
}
- @Test
Review comment:
Why this?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+ if (key == null) {
Review comment:
as above. (also, this code seems to be duplicated; we should move it into `process()` before we call the the actual `processReverse` or `processInOrder` methods.
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
Review comment:
we should not use this annotation (even if we still have code that used it... we are working on migrating test away lazily). We should instead use `assertThrows` and also verify the exception error message.
Same below.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
Review comment:
why `* 2` ?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
Review comment:
as above
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
Review comment:
Why do we suppress instead of fix the issue? (or add an exception to the `suppress.xml` file if we really need it)
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForNegativeCases() {
Review comment:
This should be two test:
- `shouldNotBeEqualForDifferentTimeDifference`
- `shouldNotBeEqualForDifferentGracePeriod`
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
Review comment:
Why do we need three tests? If you want to "randomize" it, maybe just use `Random` to generate `difference` and `grace` input instead of hard coding them?
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
Review comment:
We should add a fourth batch with ts like 10K to get the windows when the second batch drops outs, too.
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForNegativeCases() {
+
+ verifyInEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(2)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyInEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(9)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
Review comment:
As above.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
Why is `endTime = Long.MAX_VALUE`? Should it not be `firstBatchTimestamp` ?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
I agree with Sophie that his check seems a little weird. We should check that either both (sessionWindows and sessionMerger) are null or not null.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
Review comment:
`firstBatchLeftWindow` -> `firstBatchLeftWindowStart`
Maybe also introduce `firstBatchLeftWindowEnd = firstBatchTimestamp`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478588602
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, the window size based on the given maximum time difference (inclusive) between
+ * records in the same window, and the given window grace period. While the window is sliding over the input data stream, a new window is
+ * created each time a record enters the sliding window or a record drops out of the sliding window.
+ * <p>
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * <ul>
+ * <li>window {@code [3000;8000]} contains [1] (created when first record enters the window)</li>
+ * <li>window {@code [4200;9200]} contains [1,2] (created when second record enters the window)</li>
+ * <li>window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)</li>
+ * <li>window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)</li>
+ * <li>window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)</li>
Review comment:
```suggestion
* <li>window {@code [7400;12400]} contains [1,2,3] (created when third record enters the window)</li>
* <li>window {@code [8001;13001]} contains [2,3] (created when the first record drops out of the window)</li>
* <li>window {@code [9201;14201]} contains [3] (created when the second record drops out of the window)</li>
```
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, the window size based on the given maximum time difference (inclusive) between
+ * records in the same window, and the given window grace period. While the window is sliding over the input data stream, a new window is
+ * created each time a record enters the sliding window or a record drops out of the sliding window.
+ * <p>
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * <ul>
+ * <li>window {@code [3000;8000]} contains [1] (created when first record enters the window)</li>
+ * <li>window {@code [4200;9200]} contains [1,2] (created when second record enters the window)</li>
+ * <li>window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)</li>
+ * <li>window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)</li>
+ * <li>window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)</li>
+ * </ul>
+ *<p>
+ * Note that while SlidingWindows are of a fixed size, as are {@link TimeWindows}, the start and end points of the window
+ * depend on when events occur in the stream (i.e., event timestamps), similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(SlidingWindows)
+ * @see CogroupedKStream#windowedBy(SlidingWindows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ private final long timeDifferenceMs;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifferenceMs, final long graceMs) {
+ this.timeDifferenceMs = timeDifferenceMs;
+ this.graceMs = graceMs;
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period. Reject out-of-order events that arrive after {@code grace}.
+ * A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size is < 0 or grace < 0, or either can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs < 0) {
+ throw new IllegalArgumentException("Window time difference must not be negative.");
+ }
+ final String msgPrefixGrace = prepareMillisCheckFailMsgPrefix(grace, "afterWindowEnd");
Review comment:
```suggestion
final String msgPrefixGrace = prepareMillisCheckFailMsgPrefix(grace, "grace");
```
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SuppressScenarioTest.java
##########
@@ -459,6 +460,89 @@ public void shouldSupportFinalResultsForTimeWindowsWithLargeJump() {
}
}
+ @Test
+ public void shouldSupportFinalResultsForSlidingWindows() {
Review comment:
Aside from join, forgetting to test new operators in front of Suppress has also been an issue. It's great to see this test here!
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
Review comment:
Is it already guaranteed that this window actually contains the current record? It doesn't look like we're checking that `endTime >= timestamp` anywhere, and it seems like the start of the range (`timestamp - 2 * windows.timeDifferenceMs()`) could give back a window that starts and ends before the current record's timestamp.
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "1", 20L);
+ inputTopic1.pipeInput("A", "1", 22L);
+ inputTopic1.pipeInput("A", "3", 15L);
+
+ inputTopic1.pipeInput("B", "2", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "2", 18L);
+ inputTopic1.pipeInput("B", "1", 19L);
+ inputTopic1.pipeInput("B", "2", 25L);
+ inputTopic1.pipeInput("B", "3", 14L);
+
+ inputTopic1.pipeInput("C", "3", 11L);
+ inputTopic1.pipeInput("C", "4", 15L);
+ inputTopic1.pipeInput("C", "1", 16L);
+ inputTopic1.pipeInput("C", "1", 21);
+ inputTopic1.pipeInput("C", "1", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+1", 20),
+ // A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+1", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+1+1", 22),
+ // A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+1+3", 20),
+ // A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1+3", 20),
+ // A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+1+1+3", 22),
+ // A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+3", 15),
+ // A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+1+1", 22),
+
+ // B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+2", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+2+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+2+2+2", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2+1", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2+1", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+1", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+2+2+2+1", 19),
+ // B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+1+2", 25),
+ // B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+2", 25),
+ // B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+2+1+2", 25),
+ // B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+2+2+2+3", 18),
+ // B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+2+2+2+1+3", 19),
+ // B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2+1+3", 19),
+ // B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2+1+3", 19),
+ // B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+2+2+3", 14),
+
+ // C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+3", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4", 15),
+ // C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+3+4", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4+1", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1", 16),
+ // C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+3+4+1", 16),
+ // C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4+1+1", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1+1", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+1", 21),
+ // C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+3+4+1+1", 21),
+ // C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1+1+1", 23),
+ // C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+1+1", 23),
+ // C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+1", 23),
+ // C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+4+1+1+1", 23),
+
+ // D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
Review comment:
The Achilles heel of implementing new KTable features has historically been that we forgot to test them in a context that required the ValueGetter to work properly, of which Join is a notable use case. I'd actually say it should be required for every KTable operator to have a test where it's the source of a Join. For stateless operators, we should test both with and without a Materialized argument on the operator.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
+ * windowed aggregations.
+ *
+ * @param windows the specification of the aggregation {@link SlidingWindows}
+ * @return an instance of {@link TimeWindowedCogroupedKStream}
+ */
+ TimeWindowedCogroupedKStream<K, VOut> windowedBy(final SlidingWindows windows);
Review comment:
Coming back to this after completing the review, I'd say the biggest advice I'd share is to avoid whitespace changes and cleanups on the side when the PR is so long already. In fact, for my own super-complex PRs, I tend to go back over the whole diff and back out anything that's not critically important, just to lighten the load on the reviewers.
Cleanups are nice to have, but it's better to keep them in their own PRs or in more trivial ones.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWinAlreadyCreated && rightWindowIsNotEmpty(rightWinAgg, timestamp)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ private boolean rightWindowIsNotEmpty(final ValueAndTimestamp<Agg> rightWinAgg, final long timestamp) {
+ return rightWinAgg != null && rightWinAgg.timestamp() > timestamp;
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window,
+ final ValueAndTimestamp<Agg> valueAndTime,
+ final K key,
+ final V value,
+ final long closeTime,
+ final long timestamp) {
Review comment:
Might not be a bad idea to have an assertion here that the timestamp is actually in the window boundaries.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
Review comment:
minor: this could be declared `final` at the assignment on line 161
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,692 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testAggregateSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 15L);
+ inputTopic.pipeInput("A", "3", 20L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+1", 10L));
+ expected.put(5L, ValueAndTimestamp.make("0+1+2", 15L));
+ expected.put(10L, ValueAndTimestamp.make("0+1+2+3", 20L));
+ expected.put(11L, ValueAndTimestamp.make("0+2+3", 20L));
+ expected.put(12L, ValueAndTimestamp.make("0+2+3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("0+3+4", 22L));
+ expected.put(20L, ValueAndTimestamp.make("0+3+4+5", 30L));
+ expected.put(21L, ValueAndTimestamp.make("0+4+5", 30L));
+ expected.put(23L, ValueAndTimestamp.make("0+5", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testReduceSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 14L);
+ inputTopic.pipeInput("A", "3", 15L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 26L);
+ inputTopic.pipeInput("A", "6", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("1", 10L));
+ expected.put(4L, ValueAndTimestamp.make("1+2", 14L));
+ expected.put(5L, ValueAndTimestamp.make("1+2+3", 15L));
+ expected.put(11L, ValueAndTimestamp.make("2+3", 15L));
+ expected.put(12L, ValueAndTimestamp.make("2+3+4", 22L));
+ expected.put(15L, ValueAndTimestamp.make("3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("4+5", 26L));
+ expected.put(20L, ValueAndTimestamp.make("4+5+6", 30L));
+ expected.put(23L, ValueAndTimestamp.make("5+6", 30L));
+ expected.put(27L, ValueAndTimestamp.make("6", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @Test
+ public void testAggregateLargeInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
Review comment:
I'd normally say we should have a test also to verify we log properly on early records, but you already opened the PR to add early record handling, so we're good.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedKStreamImpl.java
##########
@@ -61,7 +62,7 @@
Objects.requireNonNull(groupedStream, "groupedStream can't be null");
Objects.requireNonNull(aggregator, "aggregator can't be null");
groupPatterns.put((KGroupedStreamImpl<K, ?>) groupedStream,
- (Aggregator<? super K, ? super Object, VOut>) aggregator);
+ (Aggregator<? super K, ? super Object, VOut>) aggregator);
Review comment:
```suggestion
(Aggregator<? super K, ? super Object, VOut>) aggregator);
```
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
+ * windowed aggregations.
+ *
+ * @param windows the specification of the aggregation {@link SlidingWindows}
+ * @return an instance of {@link TimeWindowedCogroupedKStream}
+ */
+ TimeWindowedCogroupedKStream<K, VOut> windowedBy(final SlidingWindows windows);
Review comment:
I'm reviewing this whole PR as-is, so there's no need to do anything now, but @mjsax 's specific suggestion is beside the point. The general feedback is that this PR is too large, which it is. We shoot for under 1K, and it's the PR author's responsibility to figure out the best way to break it up.
This policy isn't just "reviewers complaining," it's an important component of ensuring AK's quality. Long PRs overwhelm any reviewer's cognitive capacity to pay attention to every detail, so oversights are more likely to slip through into the codebase, and once they're there, you're really at the mercy of the testing layers to catch them. When the oversights are very subtle, they wind up getting released and then surface as user-reported bugs. Reviewers can't guarantee to notice every problem, but our capacity to notice problems is inversely proportional to the length of the PR.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
Review comment:
Is this condition supposed to be checking whether records are "early" with respect to now? It looks like it should be:
```suggestion
if (timestamp < (observedStreamTime - windows.timeDifferenceMs())) {
```
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,692 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testAggregateSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 15L);
+ inputTopic.pipeInput("A", "3", 20L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+1", 10L));
+ expected.put(5L, ValueAndTimestamp.make("0+1+2", 15L));
+ expected.put(10L, ValueAndTimestamp.make("0+1+2+3", 20L));
+ expected.put(11L, ValueAndTimestamp.make("0+2+3", 20L));
+ expected.put(12L, ValueAndTimestamp.make("0+2+3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("0+3+4", 22L));
+ expected.put(20L, ValueAndTimestamp.make("0+3+4+5", 30L));
+ expected.put(21L, ValueAndTimestamp.make("0+4+5", 30L));
+ expected.put(23L, ValueAndTimestamp.make("0+5", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testReduceSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 14L);
+ inputTopic.pipeInput("A", "3", 15L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 26L);
+ inputTopic.pipeInput("A", "6", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("1", 10L));
+ expected.put(4L, ValueAndTimestamp.make("1+2", 14L));
+ expected.put(5L, ValueAndTimestamp.make("1+2+3", 15L));
+ expected.put(11L, ValueAndTimestamp.make("2+3", 15L));
+ expected.put(12L, ValueAndTimestamp.make("2+3+4", 22L));
+ expected.put(15L, ValueAndTimestamp.make("3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("4+5", 26L));
+ expected.put(20L, ValueAndTimestamp.make("4+5+6", 30L));
+ expected.put(23L, ValueAndTimestamp.make("5+6", 30L));
+ expected.put(27L, ValueAndTimestamp.make("6", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @Test
+ public void testAggregateLargeInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+ builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(MockInitializer.STRING_INIT, MockAggregator.toStringInstance("+"), Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()));
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput(null, "1");
+ assertThat(appender.getMessages(), hasItem("Skipping record due to null key or value. value=[1] topic=[topic] partition=[0] offset=[0]"));
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingExpiredWindowByGrace() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KStream<String, String> stream1 = builder.stream(topic, Consumed.with(Serdes.String(), Serdes.String()));
+ stream1.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(90L)))
+ .aggregate(
+ () -> "",
+ MockAggregator.toStringInstance("+"),
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()).withCachingDisabled().withLoggingDisabled()
+ )
+ .toStream()
+ .map((key, value) -> new KeyValue<>(key.toString(), value))
+ .to("output");
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("k", "100", 200L);
+ inputTopic.pipeInput("k", "0", 100L);
+ inputTopic.pipeInput("k", "1", 101L);
+ inputTopic.pipeInput("k", "2", 102L);
+ inputTopic.pipeInput("k", "3", 103L);
+ inputTopic.pipeInput("k", "4", 104L);
+ inputTopic.pipeInput("k", "5", 105L);
+ inputTopic.pipeInput("k", "6", 15L);
+
+ assertLatenessMetrics(driver, is(7.0), is(185.0), is(96.25));
+
+ assertThat(appender.getMessages(), hasItems(
+ // left window for k@100
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[1] timestamp=[100] window=[90,100] expiration=[110] streamTime=[200]",
+ // left window for k@101
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[2] timestamp=[101] window=[91,101] expiration=[110] streamTime=[200]",
+ // left window for k@102
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[3] timestamp=[102] window=[92,102] expiration=[110] streamTime=[200]",
+ // left window for k@103
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[4] timestamp=[103] window=[93,103] expiration=[110] streamTime=[200]",
+ // left window for k@104
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[5] timestamp=[104] window=[94,104] expiration=[110] streamTime=[200]",
+ // left window for k@105
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[6] timestamp=[105] window=[95,105] expiration=[110] streamTime=[200]",
+ // left window for k@15
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[7] timestamp=[15] window=[5,15] expiration=[110] streamTime=[200]"
+ ));
+ final TestOutputTopic<String, String> outputTopic =
+ driver.createOutputTopic("output", new StringDeserializer(), new StringDeserializer());
+ assertThat(outputTopic.readRecord(), equalTo(new TestRecord<>("[k@190/200]", "+100", null, 200L)));
+ assertTrue(outputTopic.isEmpty());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testAggregateRandomInput() {
Review comment:
Awesome test. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478717127
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "1", 20L);
+ inputTopic1.pipeInput("A", "1", 22L);
+ inputTopic1.pipeInput("A", "3", 15L);
+
+ inputTopic1.pipeInput("B", "2", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "2", 18L);
+ inputTopic1.pipeInput("B", "1", 19L);
+ inputTopic1.pipeInput("B", "2", 25L);
+ inputTopic1.pipeInput("B", "3", 14L);
+
+ inputTopic1.pipeInput("C", "3", 11L);
+ inputTopic1.pipeInput("C", "4", 15L);
+ inputTopic1.pipeInput("C", "1", 16L);
+ inputTopic1.pipeInput("C", "1", 21);
+ inputTopic1.pipeInput("C", "1", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+1", 20),
+ // A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+1", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+1+1", 22),
+ // A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+1+3", 20),
+ // A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1+3", 20),
+ // A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+1+1+3", 22),
+ // A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+3", 15),
+ // A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+1+1", 22),
+
+ // B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+2", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+2+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+2+2+2", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2+1", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2+1", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+1", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+2+2+2+1", 19),
+ // B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+1+2", 25),
+ // B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+2", 25),
+ // B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+2+1+2", 25),
+ // B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+2+2+2+3", 18),
+ // B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+2+2+2+1+3", 19),
+ // B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2+1+3", 19),
+ // B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2+1+3", 19),
+ // B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+2+2+3", 14),
+
+ // C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+3", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4", 15),
+ // C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+3+4", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4+1", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1", 16),
+ // C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+3+4+1", 16),
+ // C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4+1+1", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1+1", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+1", 21),
+ // C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+3+4+1+1", 21),
+ // C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1+1+1", 23),
+ // C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+1+1", 23),
+ // C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+1", 23),
+ // C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+4+1+1+1", 23),
+
+ // D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
Review comment:
Aha, so there was a good reason for it 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466029116
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
+ * windowed aggregations.
+ *
+ * @param windows the specification of the aggregation {@link SlidingWindows}
+ * @return an instance of {@link TimeWindowedCogroupedKStream}
+ */
+ TimeWindowedCogroupedKStream<K, VOut> windowedBy(final SlidingWindows windows);
Review comment:
Ack. Was just a thought.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463088582
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
Review comment:
changed to
` if (leftWinAgg.timestamp() < timestamp &&
leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
} else {
//left window just contains the current record
valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
}`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465099913
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
Review comment:
Ack. Makes sense. Might be worth to add a comment why we need an "unexpected" large retention time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478751787
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
Review comment:
I think he means, instead of declaring it once up here and then reassigning it every iteration, we can just do `final KeyValue<> next = iterator.next()` down on line 161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465380753
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
Review comment:
Just to clarify, we don't get any additional output when the stream time is advanced and older windows drop out of the grace period. We've already forwarded their final state when the last record to update that window was processed. Not sure if that's what you meant by "get the windows when the batch drops out" or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468214782
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
Review comment:
Haha, my specialty!
The distillation of this sentence is "Windows are defined based on a record's timestamp, window size, and window grace period." I think the meaning is pretty clear, so no need to change anything.
Just to point it out, there's structural ambiguity about whether the sentence is saying "a record's (timestamp, window size, window grace period)" (I.e., three properties of the record), or whether there are three top-level things that define the window. The latter was intended. I think actually inserting "the" before "window" both times would clear it up: Windows are defined based on a record's timestamp, the window size, and the window grace period."
Another note is that because the second item in the list is so long, the structure of the list gets a little lost. It would be better in this case to use the Oxford comma to clearly delineate the boundary between the second and third items.
So, although I think this is fine as-is, if you want me to break out the red pen, I'd say:
```
* Sliding Windows are defined based on a record's timestamp, the window size based on the given maximum time difference (inclusive) between
* records in the same window, and the given window grace period.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r467231847
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
Thanks. I missed the point that this trick to pass in the windowSize only works for KafkaStreams when we pass in `Serdes` object that are used as provided...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478726427
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
Review comment:
Is that possible? It's reassigned for every iteration of the `while()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463078058
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinAlreadyCreated = false;
+
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ //potentially need to change long to instant
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next = iterator.next();
+
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ final long endTime = next.key.window().end();
+ final long startTime = next.key.window().start();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ continue;
+ } else if (endTime == timestamp) {
+ leftWinExists = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ foundLeftFirst = isLeftWindow(next) ? true : false;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else {
+ rightWindowExists = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ if (latestLeftTypeWindow != null) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window, final ValueAndTimestamp<Agg> valueAndTime, final K key,
+ final V value, final long closeTime, final long timestamp) {
+ final long windowStart = window.start();
+ final long windowEnd = window.end();
+ if (windowEnd > closeTime) {
+ //get aggregate from existing window
+ final Agg oldAgg = getValueOrNull(valueAndTime);
+ //add record's value to existing aggregate
+ final Agg newAgg = aggregator.apply(key, value, oldAgg);
+
+ windowStore.put(key,
+ ValueAndTimestamp.make(newAgg, Math.max(timestamp, valueAndTime.timestamp())),
+ windowStart);
+ tupleForwarder.maybeForward(
+ new Windowed<K>(key, window),
+ newAgg,
+ sendOldValues ? oldAgg : null,
+ windowStart);
Review comment:
good catch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463063406
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
Review comment:
Yeah I think you're right that the aggregate should be in the `if()`, good catch.
We could check max timestamp, but this scenario should be covered already. Before we create a right window we do the boolean checks and since any in-order-record won't have either a `foundLeftWinFirst` or the `prevRightWinAlreadyCreated` and one of them needs to be true for the right window to get created after the `while()`.
I haven't fully thought through checking with maxTimestamp but it seems like that would work, if that way seems clearer I can alter the algorithm and run through the examples to make sure that covers everything
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#issuecomment-683916587
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r462626773
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ public final long timeDifference;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifference, final long graceMs) {
+ this.timeDifference = timeDifference;
+ this.graceMs = graceMs;
+
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs <= 0) {
+ throw new IllegalArgumentException("Window timeDifference (timeDifference) must be larger than zero.");
+ }
+ final String msgPrefixGrace = prepareMillisCheckFailMsgPrefix(grace, "afterWindowEnd");
+ final long graceMs = ApiUtils.validateMillisecondDuration(grace, msgPrefixGrace);
+ if (graceMs < 0) {
+ throw new IllegalArgumentException("Grace period must not be negative.");
+ }
+
+ return new SlidingWindows(timeDifferenceMs, graceMs);
+ }
+
+
+ public long timeDifference() {
Review comment:
nit: also rename the method `timeDifferenceMs` to be consistent with `gracePeriodMs`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link SessionWindowedCogroupedKStream} instance that can be used to perform session
Review comment:
```suggestion
* Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
```
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
Review comment:
I think it's ok to skip this; since it's a new operator, there's no old topology to be compatible with
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ public final long timeDifference;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifference, final long graceMs) {
+ this.timeDifference = timeDifference;
+ this.graceMs = graceMs;
+
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs <= 0) {
+ throw new IllegalArgumentException("Window timeDifference (timeDifference) must be larger than zero.");
+ }
+ final String msgPrefixGrace = prepareMillisCheckFailMsgPrefix(grace, "afterWindowEnd");
+ final long graceMs = ApiUtils.validateMillisecondDuration(grace, msgPrefixGrace);
+ if (graceMs < 0) {
+ throw new IllegalArgumentException("Grace period must not be negative.");
+ }
+
+ return new SlidingWindows(timeDifferenceMs, graceMs);
+ }
+
+
+ public long timeDifference() {
+ return timeDifference;
+ }
+
+
+
+
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
Review comment:
I think we can remove this suppression (and all the ones below)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ public final long timeDifference;
+
+ /** The grace period in milliseconds. */
+ private final long graceMs;
+
+ private SlidingWindows(final long timeDifference, final long graceMs) {
+ this.timeDifference = timeDifference;
+ this.graceMs = graceMs;
+
+ }
+
+ /**
+ * Return a window definition with the window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * Reject out-of-order events that arrive after {@code grace}. A window is closed when {@code stream-time > window-end + grace-period}.
+ *
+ * @param timeDifference the max time difference (inclusive) between two records in a window
+ * @param grace the grace period to admit out-of-order events to a window
+ * @return a new window definition
+ * @throws IllegalArgumentException if the specified window size or grace is zero or negative or can't be represented as {@code long milliseconds}
+ */
+ public static SlidingWindows withTimeDifferenceAndGrace(final Duration timeDifference, final Duration grace) throws IllegalArgumentException {
+ final String msgPrefixSize = prepareMillisCheckFailMsgPrefix(timeDifference, "timeDifference");
+ final long timeDifferenceMs = ApiUtils.validateMillisecondDuration(timeDifference, msgPrefixSize);
+ if (timeDifferenceMs <= 0) {
+ throw new IllegalArgumentException("Window timeDifference (timeDifference) must be larger than zero.");
+ }
+ final String msgPrefixGrace = prepareMillisCheckFailMsgPrefix(grace, "afterWindowEnd");
+ final long graceMs = ApiUtils.validateMillisecondDuration(grace, msgPrefixGrace);
+ if (graceMs < 0) {
+ throw new IllegalArgumentException("Grace period must not be negative.");
+ }
+
+ return new SlidingWindows(timeDifferenceMs, graceMs);
+ }
+
+
+ public long timeDifference() {
+ return timeDifference;
+ }
+
+
+
+
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
+ public long gracePeriodMs() {
+ return graceMs;
+ }
+
+
+ @SuppressWarnings("deprecation") // removing segments from Windows will fix this
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ final SlidingWindows that = (SlidingWindows) o;
+ return timeDifference == that.timeDifference &&
Review comment:
nit: extra space after `return`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
+
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ // new style retention: use Materialized retention and default segmentInterval
Review comment:
let's just remove this comment since it's the only style retention here (also in `SlidingWindowedCogroupedKStreamImpl`)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinAlreadyCreated = false;
+
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ //potentially need to change long to instant
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next = iterator.next();
+
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ final long endTime = next.key.window().end();
+ final long startTime = next.key.window().start();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ continue;
+ } else if (endTime == timestamp) {
+ leftWinExists = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ foundLeftFirst = isLeftWindow(next) ? true : false;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else {
+ rightWindowExists = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ if (latestLeftTypeWindow != null) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window, final ValueAndTimestamp<Agg> valueAndTime, final K key,
+ final V value, final long closeTime, final long timestamp) {
+ final long windowStart = window.start();
+ final long windowEnd = window.end();
+ if (windowEnd > closeTime) {
+ //get aggregate from existing window
+ final Agg oldAgg = getValueOrNull(valueAndTime);
+ //add record's value to existing aggregate
+ final Agg newAgg = aggregator.apply(key, value, oldAgg);
+
+ windowStore.put(key,
+ ValueAndTimestamp.make(newAgg, Math.max(timestamp, valueAndTime.timestamp())),
+ windowStart);
+ tupleForwarder.maybeForward(
+ new Windowed<K>(key, window),
+ newAgg,
+ sendOldValues ? oldAgg : null,
+ windowStart);
Review comment:
I think we need to pass in the new maximum window timestamp here, not the window start time
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
Review comment:
Can we put this condition into a method and give it a clear name to describe what this means? eg
```
private boolean rightWindowHasNotBeenCreatedAndIsNonEmpty(..) {
return !rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)
}
```
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/TimeWindowedKStreamImpl.java
##########
@@ -19,7 +19,16 @@
import org.apache.kafka.common.serialization.Serde;
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
-import org.apache.kafka.streams.kstream.*;
Review comment:
Not sure if you did this or your IDE did it automatically, but nice 👍
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+
+import java.time.Duration;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
+ *
+ * While the window is sliding over the input data stream, a new window is created each time a record enters
+ * the sliding window or a record drops out of the sliding window.
+ *
+ * Records that come after set grace period will be ignored, i.e., a window is closed when
+ * {@code stream-time > window-end + grace-period}.
+ * <p>
+ * For example, if we have a time difference of 5000ms and the following data arrives:
+ * <pre>
+ * +--------------------------------------+
+ * | key | value | time |
+ * +-----------+-------------+------------+
+ * | A | 1 | 8000 |
+ * +-----------+-------------+------------+
+ * | A | 2 | 9200 |
+ * +-----------+-------------+------------+
+ * | A | 3 | 12400 |
+ * +-----------+-------------+------------+
+ * </pre>
+ * We'd have the following 5 windows:
+ * - window {@code [3000;8000]} contains [1] (created when first record enters the window)
+ * - window {@code [4200;9200]} contains [1,2] (created when second record enters the window)
+ * - window {@code [7400;124000]} contains [1,2,3] (created when third record enters the window)
+ * - window {@code [8001;130001]} contains [2,3] (created when the first record drops out of the window)
+ * - window {@code [9201;142001]} contains [3] (created when the second record drops out of the window)
+ *
+ * Note that while SlidingWindows are of a fixed size {@link TimeWindows}, the start and end points
+ * depend on when events are processed, similar to {@link SessionWindows}.
+ * <p>
+ * For time semantics, see {@link TimestampExtractor}.
+ *
+ * @see TimeWindows
+ * @see SessionWindows
+ * @see UnlimitedWindows
+ * @see JoinWindows
+ * @see KGroupedStream#windowedBy(Windows)
+ * @see TimestampExtractor
+ */
+
+public final class SlidingWindows {
+
+
+ /** The size of the windows in milliseconds, defined by the max time difference between records. */
+ public final long timeDifference;
Review comment:
nit: call this `timeDifferenceMs` to be in sync with `graceMs`. Also it can be private
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
Review comment:
I wonder why we have to do this for `count` but not for `aggregate` and `reduce`? Is this intentional or an oversight? cc @mjsax @vvcephei @guozhangwang
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedCogroupedKStreamImpl.java
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedCogroupedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+public class SlidingWindowedCogroupedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedCogroupedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+ private final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns;
+
+ SlidingWindowedCogroupedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode,
+ final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns) {
+ super(name, null, null, subTopologySourceNodes, streamsGraphNode, builder);
+ //keySerde and valueSerde are null because there are many different groupStreams that they could be from
+ this.windows = windows;
+ this.aggregateBuilder = aggregateBuilder;
+ this.groupPatterns = groupPatterns;
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer) {
+ return aggregate(initializer, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named) {
+ return aggregate(initializer, named, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal = new MaterializedInternal<>(
+ materialized,
+ builder,
+ CogroupedKStreamImpl.AGGREGATE_NAME);
+ return aggregateBuilder.build(
+ groupPatterns,
+ initializer,
+ new NamedInternal(named),
+ materialize(materializedInternal),
+ materializedInternal.keySerde() != null ?
+ new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference())
+ : null,
+ materializedInternal.valueSerde(),
+ materializedInternal.queryableStoreName(),
+ null,
+ windows,
+ null,
+ null);
+ }
+
+ @SuppressWarnings("deprecation")
+ // continuing to support Windows#maintainMs/segmentInterval in fallback mode
Review comment:
We can remove this
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
Seems like this should have also had a check for `sessionWindows != null`, right? Can we add that as well?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
Review comment:
I was wondering what this method is actually used for so I checked out the callers of `KStreamWindowAggregate#windows`. There's a method called `extractGracePeriod` in `GraphGraceSearchUtil` where we might actually need to make a small addition to include the new sliding window processor.
I think it's for Suppression, which needs to figure out the grace period of the upstream operator since grace period doesn't get passed in directly to `suppress`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
Review comment:
Can we also name this variable a bit more clearly instead of the comment? Like `foundClosestStartTimeWindow` or something. Same with `foundFirstEndTime`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
Review comment:
nit: alignment is off by one on the parameters
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
+
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
Review comment:
You should be able to remove this suppression and comment (here and in `SlidingWindowedCogroupedKStreamImpl`)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
+
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ // new style retention: use Materialized retention and default segmentInterval
+ final long retentionPeriod = materialized.retention().toMillis();
+
+ if ((windows.timeDifference() + windows.gracePeriodMs()) > retentionPeriod) {
+ throw new IllegalArgumentException("The retention period of the window store "
+ + name + " must be no smaller than its window time difference plus the grace period."
+ + " Got time difference=[" + windows.timeDifference() + "],"
+ + " grace=[" + windows.gracePeriodMs() + "],"
+ + " retention=[" + retentionPeriod + "]");
+ }
+
+ supplier = Stores.persistentTimestampedWindowStore(
+ materialized.storeName(),
+ Duration.ofMillis(retentionPeriod),
+ Duration.ofMillis(windows.timeDifference()),
+ false
+ );
+ }
+ final StoreBuilder<TimestampedWindowStore<K, VR>> builder = Stores.timestampedWindowStoreBuilder(
+ supplier,
+ materialized.keySerde(),
+ materialized.valueSerde()
+ );
+
+ if (materialized.loggingEnabled()) {
+ builder.withLoggingEnabled(materialized.logConfig());
+ } else {
+ builder.withLoggingDisabled();
+ }
+
+ if (materialized.cachingEnabled()) {
+ builder.withCachingEnabled();
+ }
Review comment:
Can we add an `else` here with `builder.withCachingDisabled()`? It doesn't make a difference logically, it just seems easier to understand (again, also in `SlidingWindowedCogroupedKStreamImpl`)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinAlreadyCreated = false;
+
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ //potentially need to change long to instant
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next = iterator.next();
+
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ final long endTime = next.key.window().end();
+ final long startTime = next.key.window().start();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ continue;
+ } else if (endTime == timestamp) {
+ leftWinExists = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ foundLeftFirst = isLeftWindow(next) ? true : false;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else {
+ rightWindowExists = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ if (latestLeftTypeWindow != null) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window, final ValueAndTimestamp<Agg> valueAndTime, final K key,
Review comment:
nit: put each parameter on its own line
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
Review comment:
Does that make sense? In particular I feel like we're using `Exists` to mean one thing for `left/RightWindowExists`, and then we mean another thing entirely in `prevRightWindowExists`. ie `prevRightWinAlreadyCreated` is more similar to what we mean by the `left/RightWindowExists` variables
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
Review comment:
In general it's better to use a more descriptive variable name than a shorted one with a comment. It's not always possible to describe a variable exactly in a reasonable length, but I think in this case we can say `curLeftWindowAlreadyExists` or `curLeftWindowAlreadyCreated` or something
Might be better to use `AlreadyCreated` when we're specifically talking about whether or not a window already exists in the window store, and can use `Exists` when we're talking about whether a window is possible regardless of whether it currently has been created or not
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
Review comment:
this comment doesn't seem quite correct
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
Review comment:
maybe add a comment saying that this condition will only be hit on the very first record. Or it might be reasonable to pull this one condition out of the loop and just handle it before entering the loop
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
Review comment:
Seems like we're aggregating with the new value twice; we call `aggregator.apply` once in this if/else branch but then also call it again in `putAndForward`, right?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
Review comment:
Should this be inside the `if (!foundFirst)` condition above? We only want to save the aggregate of the first window we find with a start time less than the timestamp right?
Also, I think we might need to check that the max timestamp of this window is greater than the current record's timestamp. If not, then the right window will be empty.
For example, we have a record A at 10 and a record B at 11 and then process a record at 15. Obviously, the new right window will be empty. But the first window we'll find with a start time less than 15 will be [11, 21] with agg B.
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
Review comment:
Actually maybe `foundRight/LeftWindowAggregate` would be good, since that's what "the window with the closest start/end time to the record" actually means to us
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
Review comment:
Can we remove this and just `break` out of the loop immediately at the end of the `isLeftWindow` condition block?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
Review comment:
Since it's a left window, the max timestamp should always be `timestamp`, right?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
+
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ // new style retention: use Materialized retention and default segmentInterval
+ final long retentionPeriod = materialized.retention().toMillis();
+
+ if ((windows.timeDifference() + windows.gracePeriodMs()) > retentionPeriod) {
Review comment:
Do you think we actually need to enforce that the retention period be a little longer for sliding windows? I was just thinking that since the range scan starts at `timestamp - 2 * windows.timeDifference()`, maybe we should actually enforce that the retention period be >= `2 * timeDifference + gracePeriod` in case we need to get the aggregate value from some older window that has technically expired.
Haven't checked the math so I'm not sure that's the correct value exactly, but it seems like it might need to be a little bigger. Any thoughts?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
Review comment:
Can we add a comment to clarify that we're checking whether it's a left window because that tells us there was a record at this window's end time
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
Review comment:
Ok I may have lost the trail of logic here...are we just checking `prevRightWinExists` as an indicator of whether we actually found any records to the left of our record within range? Could/should we check `foundFirstEndTime` instead?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
Review comment:
I feel like I'm just way overthinking this, but I keep getting these variables confused. Maybe we could call this guy `prevRightWindowCanExist`? Does that seem to get at its underlying purpose?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465369865
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
@mjsax why do we have a single method that accepts all three window types and then checks them all individually to enforce that only one type of window is actually "set"? Seems like we could enforce this implicitly by having a separate method for time, session, and non-windowed aggregates and then just calling the correct signature. ie `SessionWindowedCogroupedKStreamImpl` calls `build(...SessionWindows, SessionMerger) and so on.
Maybe I'm missing something here because I wasn't following the cogroup KIP that closely, but is this even exposed to the user in any way? My understanding is that there's no way for this check to be violated by any kind of user input, because this method is only ever called directly by Streams internal code with `null` hardcoded for the unused window types.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466051779
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long endTime = next.key.window().end();
Review comment:
Ok cool, just checking. Either `endTime = window.start + timeDifferenceMs` or `endTime = window.end` is fine as long as we're consistent (I guess we only define it in two places, the forward and reverse algorithms?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466032331
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
We pass in all parameter to sharing to code that creates the `StatefulProcessorNode` -- not sure if it's the best way to structure the code and I am happy to split it up into multiple methods call (as long as we avoid code duplication). And yes, you are right, it's internal and the checks are just for us to avoid programming errors. Users should never be exposed to it. I personally tend to make a lot of mistakes and the more checks we have in place the better IMHO :)
If @lct45 want's she can just do a side cleanup PR to fix it, and rebase this PR after the cleanup PR was merged? Or we do it as follow up. Whatever works best for you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466063583
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
No she's right, this problem is not resolved at all. You can pass in `windowSize` to the constructor for `TimeWindowedDeserializer` all you want but it just gets ignored because the actual deserializer object you instantiate is thrown away. Whether you're reading in records through a Java Consumer or the console consumer (for some reason this test does both), the actual deserializer is always constructed within the consumer based on the configs. There's a config for the windowed inner class which is properly set in `TimeWindowedDeserializer#configure` but no config for the `windowSize` so there's no way to set it at the moment.
tl;dr there's no point in having serde constructors accept parameters, they need to be set through `configure`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466033381
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
Review comment:
I am not a native speaker... Don't ask me... Mr.John should know -- he has the proper education for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r472436400
##########
File path: checkstyle/suppressions.xml
##########
@@ -167,6 +167,9 @@
<suppress checks="(FinalLocalVariable|UnnecessaryParentheses|BooleanExpressionComplexity|CyclomaticComplexity|WhitespaceAfter|LocalVariableName)"
files="Murmur3.java"/>
+ <suppress checks="(CyclomaticComplexity)"
+ files="CogroupedStreamAggregateBuilder.java"/>
Review comment:
We don't!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465369865
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
@mjsax why do we have a single method that accepts all three window types and then checks them all individually to enforce that only one type of window is actually "set"? Seems like we could enforce this implicitly by having a separate method for time, session, and non-windowed aggregates and then just calling the correct signature. ie `SessionWindowedCogroupedKStreamImpl` calls `build(...SessionWindows, SessionMerger) and so on.
Maybe I'm missing something here because I wasn't following the cogroup KIP that closely, but is this even exposed to the user in any way? My understanding is that there's no way for this check to be violated by any kind of user input, because this method is only ever called directly by Streams internal code with `null` hardcoded for the unused window types. I think it's more of an internal consistency check for Streams than an input validation for the user (and it seems unnecessary: see above)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r475919413
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testAggregateSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 15L);
+ inputTopic.pipeInput("A", "3", 20L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+1", 10L));
+ expected.put(5L, ValueAndTimestamp.make("0+1+2", 15L));
+ expected.put(10L, ValueAndTimestamp.make("0+1+2+3", 20L));
+ expected.put(11L, ValueAndTimestamp.make("0+2+3", 20L));
+ expected.put(12L, ValueAndTimestamp.make("0+2+3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("0+3+4", 22L));
+ expected.put(20L, ValueAndTimestamp.make("0+3+4+5", 30L));
+ expected.put(21L, ValueAndTimestamp.make("0+4+5", 30L));
+ expected.put(23L, ValueAndTimestamp.make("0+5", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testReduceSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 14L);
+ inputTopic.pipeInput("A", "3", 15L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 26L);
+ inputTopic.pipeInput("A", "6", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("1", 10L));
+ expected.put(4L, ValueAndTimestamp.make("1+2", 14L));
+ expected.put(5L, ValueAndTimestamp.make("1+2+3", 15L));
+ expected.put(11L, ValueAndTimestamp.make("2+3", 15L));
+ expected.put(12L, ValueAndTimestamp.make("2+3+4", 22L));
+ expected.put(15L, ValueAndTimestamp.make("3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("4+5", 26L));
+ expected.put(20L, ValueAndTimestamp.make("4+5+6", 30L));
+ expected.put(23L, ValueAndTimestamp.make("5+6", 30L));
+ expected.put(27L, ValueAndTimestamp.make("6", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @Test
+ public void testAggregateLargeInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+ builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(MockInitializer.STRING_INIT, MockAggregator.toStringInstance("+"), Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()));
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput(null, "1");
+ assertThat(appender.getMessages(), hasItem("Skipping record due to null key or value. value=[1] topic=[topic] partition=[0] offset=[0]"));
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingExpiredWindowByGrace() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KStream<String, String> stream1 = builder.stream(topic, Consumed.with(Serdes.String(), Serdes.String()));
+ stream1.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(90L)))
+ .aggregate(
+ () -> "",
+ MockAggregator.toStringInstance("+"),
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()).withCachingDisabled().withLoggingDisabled()
+ )
+ .toStream()
+ .map((key, value) -> new KeyValue<>(key.toString(), value))
+ .to("output");
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("k", "100", 200L);
+ inputTopic.pipeInput("k", "0", 100L);
+ inputTopic.pipeInput("k", "1", 101L);
+ inputTopic.pipeInput("k", "2", 102L);
+ inputTopic.pipeInput("k", "3", 103L);
+ inputTopic.pipeInput("k", "4", 104L);
+ inputTopic.pipeInput("k", "5", 105L);
+ inputTopic.pipeInput("k", "6", 15L);
+
+ assertLatenessMetrics(driver, is(7.0), is(185.0), is(96.25));
+
+ assertThat(appender.getMessages(), hasItems(
+ // left window for k@100
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[1] timestamp=[100] window=[90,100] expiration=[110] streamTime=[200]",
+ // left window for k@101
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[2] timestamp=[101] window=[91,101] expiration=[110] streamTime=[200]",
+ // left window for k@102
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[3] timestamp=[102] window=[92,102] expiration=[110] streamTime=[200]",
+ // left window for k@103
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[4] timestamp=[103] window=[93,103] expiration=[110] streamTime=[200]",
+ // left window for k@104
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[5] timestamp=[104] window=[94,104] expiration=[110] streamTime=[200]",
+ // left window for k@105
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[6] timestamp=[105] window=[95,105] expiration=[110] streamTime=[200]",
+ // left window for k@15
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[7] timestamp=[15] window=[5,15] expiration=[110] streamTime=[200]"
+ ));
+ final TestOutputTopic<String, String> outputTopic =
+ driver.createOutputTopic("output", new StringDeserializer(), new StringDeserializer());
+ assertThat(outputTopic.readRecord(), equalTo(new TestRecord<>("[k@190/200]", "+100", null, 200L)));
+ assertTrue(outputTopic.isEmpty());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testProcessorRandomInput() {
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(10000)))
+ .aggregate(
+ () -> "",
+ (key, value, aggregate) -> {
+ aggregate += value;
+ final char[] ch = aggregate.toCharArray();
+ Arrays.sort(ch);
+ aggregate = String.valueOf(ch);
+ return aggregate;
+ },
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ final long seed = new Random().nextLong();
+ final Random shuffle = new Random(seed);
+
+ try {
+
+ final List<ValueAndTimestamp<String>> input = Arrays.asList(
+ ValueAndTimestamp.make("A", 10L),
+ ValueAndTimestamp.make("B", 15L),
+ ValueAndTimestamp.make("C", 16L),
+ ValueAndTimestamp.make("D", 18L),
+ ValueAndTimestamp.make("E", 30L),
+ ValueAndTimestamp.make("F", 40L),
+ ValueAndTimestamp.make("G", 55L),
+ ValueAndTimestamp.make("H", 56L),
+ ValueAndTimestamp.make("I", 58L),
+ ValueAndTimestamp.make("J", 58L),
+ ValueAndTimestamp.make("K", 62L),
+ ValueAndTimestamp.make("L", 63L),
+ ValueAndTimestamp.make("M", 63L),
+ ValueAndTimestamp.make("N", 63L),
+ ValueAndTimestamp.make("O", 76L),
+ ValueAndTimestamp.make("P", 77L),
+ ValueAndTimestamp.make("Q", 80L)
+ );
+
+ Collections.shuffle(input, shuffle);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ for (int i = 0; i < input.size(); i++) {
+ inputTopic1.pipeInput("A", input.get(i).value(), input.get(i).timestamp());
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> results = new HashMap<>();
+
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (results.putIfAbsent(start, valueAndTimestamp) != null) {
+ results.replace(start, valueAndTimestamp);
+ }
+ }
+ randomEqualityCheck(results, seed);
+ } catch (final AssertionError t) {
+ throw new AssertionError(
+ "Assertion failed in randomized test. Reproduce with seed: " + seed + ".",
+ t
+ );
+ } catch (final Throwable t) {
+ final StringBuilder sb =
+ new StringBuilder()
+ .append("Exception in randomized scenario. Reproduce with seed: ")
+ .append(seed)
+ .append(".");
+ throw new AssertionError(sb.toString(), t);
+ }
+ }
+
+ private void randomEqualityCheck(final Map<Long, ValueAndTimestamp<String>> actual, final Long seed) {
Review comment:
nit: can you call this something a bit more direct, eg `verifyRandomTestResults` ?
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testAggregateSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 15L);
+ inputTopic.pipeInput("A", "3", 20L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+1", 10L));
+ expected.put(5L, ValueAndTimestamp.make("0+1+2", 15L));
+ expected.put(10L, ValueAndTimestamp.make("0+1+2+3", 20L));
+ expected.put(11L, ValueAndTimestamp.make("0+2+3", 20L));
+ expected.put(12L, ValueAndTimestamp.make("0+2+3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("0+3+4", 22L));
+ expected.put(20L, ValueAndTimestamp.make("0+3+4+5", 30L));
+ expected.put(21L, ValueAndTimestamp.make("0+4+5", 30L));
+ expected.put(23L, ValueAndTimestamp.make("0+5", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testReduceSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 14L);
+ inputTopic.pipeInput("A", "3", 15L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 26L);
+ inputTopic.pipeInput("A", "6", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("1", 10L));
+ expected.put(4L, ValueAndTimestamp.make("1+2", 14L));
+ expected.put(5L, ValueAndTimestamp.make("1+2+3", 15L));
+ expected.put(11L, ValueAndTimestamp.make("2+3", 15L));
+ expected.put(12L, ValueAndTimestamp.make("2+3+4", 22L));
+ expected.put(15L, ValueAndTimestamp.make("3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("4+5", 26L));
+ expected.put(20L, ValueAndTimestamp.make("4+5+6", 30L));
+ expected.put(23L, ValueAndTimestamp.make("5+6", 30L));
+ expected.put(27L, ValueAndTimestamp.make("6", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @Test
+ public void testAggregateLargeInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+ builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(MockInitializer.STRING_INIT, MockAggregator.toStringInstance("+"), Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()));
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput(null, "1");
+ assertThat(appender.getMessages(), hasItem("Skipping record due to null key or value. value=[1] topic=[topic] partition=[0] offset=[0]"));
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingExpiredWindowByGrace() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KStream<String, String> stream1 = builder.stream(topic, Consumed.with(Serdes.String(), Serdes.String()));
+ stream1.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(90L)))
+ .aggregate(
+ () -> "",
+ MockAggregator.toStringInstance("+"),
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()).withCachingDisabled().withLoggingDisabled()
+ )
+ .toStream()
+ .map((key, value) -> new KeyValue<>(key.toString(), value))
+ .to("output");
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("k", "100", 200L);
+ inputTopic.pipeInput("k", "0", 100L);
+ inputTopic.pipeInput("k", "1", 101L);
+ inputTopic.pipeInput("k", "2", 102L);
+ inputTopic.pipeInput("k", "3", 103L);
+ inputTopic.pipeInput("k", "4", 104L);
+ inputTopic.pipeInput("k", "5", 105L);
+ inputTopic.pipeInput("k", "6", 15L);
+
+ assertLatenessMetrics(driver, is(7.0), is(185.0), is(96.25));
+
+ assertThat(appender.getMessages(), hasItems(
+ // left window for k@100
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[1] timestamp=[100] window=[90,100] expiration=[110] streamTime=[200]",
+ // left window for k@101
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[2] timestamp=[101] window=[91,101] expiration=[110] streamTime=[200]",
+ // left window for k@102
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[3] timestamp=[102] window=[92,102] expiration=[110] streamTime=[200]",
+ // left window for k@103
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[4] timestamp=[103] window=[93,103] expiration=[110] streamTime=[200]",
+ // left window for k@104
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[5] timestamp=[104] window=[94,104] expiration=[110] streamTime=[200]",
+ // left window for k@105
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[6] timestamp=[105] window=[95,105] expiration=[110] streamTime=[200]",
+ // left window for k@15
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[7] timestamp=[15] window=[5,15] expiration=[110] streamTime=[200]"
+ ));
+ final TestOutputTopic<String, String> outputTopic =
+ driver.createOutputTopic("output", new StringDeserializer(), new StringDeserializer());
+ assertThat(outputTopic.readRecord(), equalTo(new TestRecord<>("[k@190/200]", "+100", null, 200L)));
+ assertTrue(outputTopic.isEmpty());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testProcessorRandomInput() {
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(10000)))
+ .aggregate(
+ () -> "",
+ (key, value, aggregate) -> {
+ aggregate += value;
+ final char[] ch = aggregate.toCharArray();
+ Arrays.sort(ch);
+ aggregate = String.valueOf(ch);
+ return aggregate;
Review comment:
Can you leave a brief comment here explaining why we're doing something slightly more complicated in the aggregator for this test
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testAggregateSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 15L);
+ inputTopic.pipeInput("A", "3", 20L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+1", 10L));
+ expected.put(5L, ValueAndTimestamp.make("0+1+2", 15L));
+ expected.put(10L, ValueAndTimestamp.make("0+1+2+3", 20L));
+ expected.put(11L, ValueAndTimestamp.make("0+2+3", 20L));
+ expected.put(12L, ValueAndTimestamp.make("0+2+3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("0+3+4", 22L));
+ expected.put(20L, ValueAndTimestamp.make("0+3+4+5", 30L));
+ expected.put(21L, ValueAndTimestamp.make("0+4+5", 30L));
+ expected.put(23L, ValueAndTimestamp.make("0+5", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testReduceSmallInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("A", "1", 10L);
+ inputTopic.pipeInput("A", "2", 14L);
+ inputTopic.pipeInput("A", "3", 15L);
+ inputTopic.pipeInput("A", "4", 22L);
+ inputTopic.pipeInput("A", "5", 26L);
+ inputTopic.pipeInput("A", "6", 30L);
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> actual = new HashMap<>();
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (actual.putIfAbsent(start, valueAndTimestamp) != null) {
+ actual.replace(start, valueAndTimestamp);
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("1", 10L));
+ expected.put(4L, ValueAndTimestamp.make("1+2", 14L));
+ expected.put(5L, ValueAndTimestamp.make("1+2+3", 15L));
+ expected.put(11L, ValueAndTimestamp.make("2+3", 15L));
+ expected.put(12L, ValueAndTimestamp.make("2+3+4", 22L));
+ expected.put(15L, ValueAndTimestamp.make("3+4", 22L));
+ expected.put(16L, ValueAndTimestamp.make("4+5", 26L));
+ expected.put(20L, ValueAndTimestamp.make("4+5+6", 30L));
+ expected.put(23L, ValueAndTimestamp.make("5+6", 30L));
+ expected.put(27L, ValueAndTimestamp.make("6", 30L));
+
+ assertEquals(expected, actual);
+ }
+
+ @Test
+ public void testAggregateLargeInput() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+ builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(MockInitializer.STRING_INIT, MockAggregator.toStringInstance("+"), Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()));
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput(null, "1");
+ assertThat(appender.getMessages(), hasItem("Skipping record due to null key or value. value=[1] topic=[topic] partition=[0] offset=[0]"));
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingExpiredWindowByGrace() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KStream<String, String> stream1 = builder.stream(topic, Consumed.with(Serdes.String(), Serdes.String()));
+ stream1.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(90L)))
+ .aggregate(
+ () -> "",
+ MockAggregator.toStringInstance("+"),
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()).withCachingDisabled().withLoggingDisabled()
+ )
+ .toStream()
+ .map((key, value) -> new KeyValue<>(key.toString(), value))
+ .to("output");
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("k", "100", 200L);
+ inputTopic.pipeInput("k", "0", 100L);
+ inputTopic.pipeInput("k", "1", 101L);
+ inputTopic.pipeInput("k", "2", 102L);
+ inputTopic.pipeInput("k", "3", 103L);
+ inputTopic.pipeInput("k", "4", 104L);
+ inputTopic.pipeInput("k", "5", 105L);
+ inputTopic.pipeInput("k", "6", 15L);
+
+ assertLatenessMetrics(driver, is(7.0), is(185.0), is(96.25));
+
+ assertThat(appender.getMessages(), hasItems(
+ // left window for k@100
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[1] timestamp=[100] window=[90,100] expiration=[110] streamTime=[200]",
+ // left window for k@101
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[2] timestamp=[101] window=[91,101] expiration=[110] streamTime=[200]",
+ // left window for k@102
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[3] timestamp=[102] window=[92,102] expiration=[110] streamTime=[200]",
+ // left window for k@103
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[4] timestamp=[103] window=[93,103] expiration=[110] streamTime=[200]",
+ // left window for k@104
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[5] timestamp=[104] window=[94,104] expiration=[110] streamTime=[200]",
+ // left window for k@105
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[6] timestamp=[105] window=[95,105] expiration=[110] streamTime=[200]",
+ // left window for k@15
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[7] timestamp=[15] window=[5,15] expiration=[110] streamTime=[200]"
+ ));
+ final TestOutputTopic<String, String> outputTopic =
+ driver.createOutputTopic("output", new StringDeserializer(), new StringDeserializer());
+ assertThat(outputTopic.readRecord(), equalTo(new TestRecord<>("[k@190/200]", "+100", null, 200L)));
+ assertTrue(outputTopic.isEmpty());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testProcessorRandomInput() {
Review comment:
nit: `testAggregateRandomInput` to match up with other test names
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r480429410
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
Review comment:
Thanks for the confirmation! I agree with your thinking.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478751787
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
Review comment:
I think he means, instead of declaring it once up here and then reassigning it every iteration, we can just do `final KeyValue<> next = iterator.next()` down on line 161. We don't need it outside the loop
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463073842
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
Review comment:
Aha, I thought there was a scenario where the `putAndForward` function wouldn't be so simple. Yeah you're right, I updated it so the value of the record is only added in `putAndForward`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465336976
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
+ Instant.ofEpochMilli(timestamp + 1))
Review comment:
So we can check to see if the record that is being processed has an already existing right window (that would start at timestamp+1) without doing another call to the store
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465101059
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
+
+ if ((windows.timeDifferenceMs() * 2 + windows.gracePeriodMs()) > retentionPeriod) {
Review comment:
@lct45 With regard to above: the error message to does align to the required minimum retention time. It says `must be no smaller than its window time difference plus the grace period.`...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466070317
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
Created a ticket for this here: https://issues.apache.org/jira/browse/KAFKA-10366 let me know if the description isn't clear
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465382189
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
Review comment:
Not sure, I think `and given window grace` makes grammatical sense. But either way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463083346
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
Review comment:
Yeah that's what `prevRightWinExists` is doing right now, we could store the full `valueAndTimestamp` for `foundFirstEndTime` and check to see if the max timestamp is within the range of recordTime-timeDifference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465348679
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForNegativeCases() {
+
+ verifyInEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(2)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyInEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(9)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+
+ verifyInEquality(
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(4), ofMillis(2)),
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(2))
+ );
+
+ assertNotEquals(
Review comment:
They look to be fairly similar, and it seems like the tests use both consistently. `verifyInEquality` seems to be more thorough, and to be consistent with the above `equalsAndHashcodeShouldBeValidForPositiveCases` I think I'll use `verifyInEquality` for this test, unless someone has an objection
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465379566
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifferenceMs()),
+ Instant.ofEpochMilli(timestamp + 1))
Review comment:
Maybe we can add this answer as a comment in the code for future readers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r464728901
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
+ * windowed aggregations.
+ *
+ * @param windows the specification of the aggregation {@link SlidingWindows}
+ * @return an instance of {@link TimeWindowedCogroupedKStream}
+ */
+ TimeWindowedCogroupedKStream<K, VOut> windowedBy(final SlidingWindows windows);
Review comment:
This PR is rather larger. Would it maybe make sense to split it into 2 and add co-group in it's own PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r479344784
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
Review comment:
Thanks. From the other thread, it sounds like I misunderstood `putAndForward` as adding the `value` to the `window`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468653951
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
Review comment:
I pulled it out for all except one, because there's one call to `windowStore` that returns a `<Windowed<String>, Long>` and the other calls to `windowStore` return a `<Windowed<String>, String>`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r472440045
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L))))); }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER, Materialized.as("store")));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ null,
+ Materialized.as("store")));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, MockAggregator.TOSTRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null, Materialized.as("store")));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfNamedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Named) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnCountIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.count((Materialized<String, Long, WindowStore<Bytes, byte[]>>) null));
+ }
+
+ @Test
+ public void shouldThrowIllegalArgumentWhenRetentionIsTooSmall() {
+ assertThrows(IllegalArgumentException.class, () -> windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized
+ .<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withRetention(ofMillis(1L))
+ )
+ );
+ }
+
+ @Test
+ public void ShouldDropWindowsOutsideOfRetention() {
+ final WindowBytesStoreSupplier storeSupplies = Stores.inMemoryWindowStore("aggregated", ofMillis(1200L), ofMillis(100L), false);
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String>as(storeSupplies)
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withCachingDisabled());
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(TOPIC, new StringSerializer(), new StringSerializer());
+
+ inputTopic.pipeInput("1", "2", 100L);
+ inputTopic.pipeInput("1", "3", 500L);
+ inputTopic.pipeInput("1", "4", 1000L);
+ inputTopic.pipeInput("1", "5", 2000L);
+
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "1", ofEpochMilli(0), ofEpochMilli(10000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(900, 1000)), "0+4"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(1900, 2000)), "0+5"))));
Review comment:
That `fetch` only returns a `WindowStoreIterator` instead of a `KeyValueIterator`, which I don't think is a huge deal but we wouldn't get the start/end time of the window which is nice to have for the test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463077196
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
Review comment:
100%
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463091437
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
Review comment:
To go with your above comment about the maxTimestamp, I changed this to be `if (!rightWinAlreadyCreated && rightWinAgg.timestamp() > timestamp)` is this clearer or should it still be in a new method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478730796
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWinAlreadyCreated && rightWindowIsNotEmpty(rightWinAgg, timestamp)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ private boolean rightWindowIsNotEmpty(final ValueAndTimestamp<Agg> rightWinAgg, final long timestamp) {
+ return rightWinAgg != null && rightWinAgg.timestamp() > timestamp;
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window,
+ final ValueAndTimestamp<Agg> valueAndTime,
+ final K key,
+ final V value,
+ final long closeTime,
+ final long timestamp) {
Review comment:
I don't think that would be true all the time, since the current record's right window wouldn't contain the current record and is created through this method. If it helps for clarity, I can add a check that if we're _not_ creating the right window then the timestamp needs to be within the window, and otherwise confirm that we're creating the right window
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465756356
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long endTime = next.key.window().end();
Review comment:
Nope, since we aren't storing truncated windows
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465194845
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
Review comment:
Re-examining the test, it looks like it does the same thing as `gracePeriodMustNotBeNegative()` so I think the test can be removed entirely
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465111665
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
Review comment:
Good catch, will do
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#issuecomment-679394512
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465094963
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
Because the windows are sorted, the windows created by each record aren't consecutive, so I added comments describing each window, but only did it for A since all the other keys are processed the exact same way. Sample comment: `// A @ secondBatchTimestamp left window created when A @ secondBatchTimestamp processed`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465399853
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,19 +135,25 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
-
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null && sessionMerger == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null && sessionMerger == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
- kStreamAggregate = new KStreamSessionWindowAggregate<>(sessionWindows, storeBuilder.name(), initializer, aggregator, sessionMerger);
+ } else if (windows == null && slidingWindows != null && sessionWindows == null && sessionMerger == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionWindows != null) {
+ if (sessionMerger == null) {
+ throw new IllegalArgumentException("sessionMerger cannot be null for sessionWindows");
+ } else {
+ kStreamAggregate = new KStreamSessionWindowAggregate<>(sessionWindows, storeBuilder.name(), initializer, aggregator, sessionMerger);
+ }
} else {
- throw new IllegalArgumentException("must include windows OR sessionWindows + sessionMerger OR all must be null");
+ throw new IllegalArgumentException("must include windows OR slidingWindows OR sessionWindows + sessionMerger OR all must be null");
Review comment:
Technically this is an `XOR` not an `OR` 😛
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedCogroupedKStreamImpl.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedCogroupedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+public class SlidingWindowedCogroupedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedCogroupedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+ private final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns;
+
+ SlidingWindowedCogroupedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode,
+ final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns) {
+ super(name, null, null, subTopologySourceNodes, streamsGraphNode, builder);
+ //keySerde and valueSerde are null because there are many different groupStreams that they could be from
+ this.windows = windows;
+ this.aggregateBuilder = aggregateBuilder;
+ this.groupPatterns = groupPatterns;
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer) {
+ return aggregate(initializer, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named) {
+ return aggregate(initializer, named, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal = new MaterializedInternal<>(
+ materialized,
+ builder,
+ CogroupedKStreamImpl.AGGREGATE_NAME);
+ return aggregateBuilder.build(
+ groupPatterns,
+ initializer,
+ new NamedInternal(named),
+ materialize(materializedInternal),
+ materializedInternal.keySerde() != null ?
+ new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs())
+ : null,
+ materializedInternal.valueSerde(),
+ materializedInternal.queryableStoreName(),
+ null,
+ windows,
+ null,
+ null);
+ }
+
+ private StoreBuilder<TimestampedWindowStore<K, V>> materialize(
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention().toMillis();
Review comment:
I think we need a null check here like we have down in `SlidingWindowedKStreamImpl#materialize`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedCogroupedKStreamImpl.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedCogroupedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+public class SlidingWindowedCogroupedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedCogroupedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+ private final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns;
+
+ SlidingWindowedCogroupedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final CogroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode,
+ final Map<KGroupedStreamImpl<K, ?>, Aggregator<? super K, ? super Object, V>> groupPatterns) {
+ super(name, null, null, subTopologySourceNodes, streamsGraphNode, builder);
+ //keySerde and valueSerde are null because there are many different groupStreams that they could be from
+ this.windows = windows;
+ this.aggregateBuilder = aggregateBuilder;
+ this.groupPatterns = groupPatterns;
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer) {
+ return aggregate(initializer, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named) {
+ return aggregate(initializer, named, Materialized.with(null, null));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> aggregate(final Initializer<V> initializer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal = new MaterializedInternal<>(
+ materialized,
+ builder,
+ CogroupedKStreamImpl.AGGREGATE_NAME);
+ return aggregateBuilder.build(
+ groupPatterns,
+ initializer,
+ new NamedInternal(named),
+ materialize(materializedInternal),
+ materializedInternal.keySerde() != null ?
+ new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs())
+ : null,
+ materializedInternal.valueSerde(),
+ materializedInternal.queryableStoreName(),
+ null,
+ windows,
+ null,
+ null);
+ }
+
+ private StoreBuilder<TimestampedWindowStore<K, V>> materialize(
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention().toMillis();
+
+ if ((windows.timeDifferenceMs() + windows.gracePeriodMs()) > retentionPeriod) {
Review comment:
Think you missed changing this in the cogrouped class, this should be 2*timeDifference right?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
+
+ if ((windows.timeDifferenceMs() * 2 + windows.gracePeriodMs()) > retentionPeriod) {
+ throw new IllegalArgumentException("The retention period of the window store "
+ + name + " must be no smaller than its window time difference plus the grace period."
Review comment:
Should be `no smaller than twice its window time difference...`
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
Review comment:
Can you leave a TODO here to make sure we remember to change this to `reverseFetch`? Seems unlikely we'd forget, but you never know
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /*
Review comment:
extra `/*` here
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
Review comment:
Just a note to other reviewers: we're planning to revisit the issue of "early" records later and are just dropping them for now to make the general algorithm easier to review and understand. It needs some special handling for the edge case of records that arrive earlier than the full sliding window due to the inability to store windows with negative start times
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
Review comment:
We don't technically need `continue` at the end of each condition, right?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
Review comment:
Can we remove the `Math.max` thing for now and just drop records that are too early for us to process for now?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamWindowAggregate.java
##########
@@ -120,7 +120,6 @@ public void process(final K key, final V value) {
final long timestamp = context().timestamp();
observedStreamTime = Math.max(observedStreamTime, timestamp);
final long closeTime = observedStreamTime - windows.gracePeriodMs();
-
Review comment:
can you revert the line changes here and below? Nothing wrong with them, but the fewer lines/classes changed in the PR, the better
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
Review comment:
```suggestion
// keep the left type window closest to the record
```
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long endTime = next.key.window().end();
+ final long startTime = next.key.window().start();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ continue;
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
Review comment:
Yeah especially since we use the same condition for both the forward and reverse case, let's just pull the `rightWinAgg != null && rightWinAgg.timestamp() > timestamp` out into a separate method
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
+
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForNegativeCases() {
Review comment:
Took me a second to understand this test, "NegativeCases" made me think the timeDifference/grace were supposed to be negative. +1 to Matthias's suggestion for naming (and splitting into two tests)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
Review comment:
> unless we also move a check for if the first is a window whose aggregate we need to update
Which check? I was just thinking that, since the window starting at record.timestamp + 1 is basically a special case, we can just pull it out of the loop completely. We don't have to update anything since the record doesn't fall into this window, right? Basically just before entering the loop we check `if next.key.window().start() == timestamp + 1` and if so set `rightWinAlreadyCreated` and then skip to the next record
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
Review comment:
comment doesn't seem to match the query bounds (missing a +1?)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long endTime = next.key.window().end();
Review comment:
Is there any reason this wouldn't just be `window.start + timeDifferenceMs`?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
Review comment:
nit: put this on one line
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
+ }
+
+ @Test
+ public void shouldSetGracePeriod() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(ANY_SIZE)).gracePeriodMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void graceMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(-1));
+ }
+
+ @Test
+ public void gracePeriodShouldEnforceBoundaries() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(0L));
+
+ try {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3L), ofMillis(-1L));
+ fail("should not accept negatives");
+ } catch (final IllegalArgumentException e) {
+ //expected
+ }
+ }
+
+ @Test
+ public void equalsAndHashcodeShouldBeValidForPositiveCases() {
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(3)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(1)));
+
+ verifyEquality(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)), SlidingWindows.withTimeDifferenceAndGrace(ofMillis(3), ofMillis(4)));
Review comment:
+1 to using a random number instead of multiple lines. If it does happen to fail on a specific random number, we should be sure to print that number for reproducing it later. See TaskAssignorConvergenceTest#runRandomizedScenario for example
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundFirst) {
+ foundFirst = true;
+ if (isLeftWindow(next)) {
+ foundLeftFirst = true;
+ }
+ }
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinExists = true;
+ continue;
+ } else {
+ if (!foundFirstEndTime) {
+ leftWinAgg = next.value;
+ foundFirstEndTime = true;
+ }
+ if (prevRightWinExists) {
+ break;
+ }
+ if (isLeftWindow(next)) {
+ prevRightWinExists = true;
+ final long rightWinStart = next.key.window().end() + 1;
+ if (windowStartTimes.contains(rightWinStart)) {
+ prevRightWinAlreadyCreated = true;
+ } else {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifference);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+ }
+ }
+ }
+
+ //create the left window of the current record if it's not created
+ if (!leftWinExists) {
+ final Agg aggValue;
+ final long newTimestamp;
+ //confirms that the left window contains more than the current record
+ if (prevRightWinExists) {
+ aggValue = aggregator.apply(key, value, getValueOrNull(leftWinAgg));
+ newTimestamp = leftWinAgg.timestamp();
+ } else {
+ //left window just contains the current record
+ aggValue = aggregator.apply(key, value, initializer.apply());
+ newTimestamp = timestamp;
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifference, timestamp);
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(aggValue, Math.max(timestamp, newTimestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for
+ if (!rightWindowExists && (foundLeftFirst || prevRightWinAlreadyCreated)) {
Review comment:
Awesome! I might still recommend pulling the `rightWinAgg != null && rightWinAgg.timestamp() > timestamp` check out into a method called `rightWindowIsNonEmpty` or something, but it's definitely a lot easier to understand now even without that 😄
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifferenceMs()) : null,
+ materializedInternal.valueSerde());
+ }
+
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ final long retentionPeriod = materialized.retention() != null ? materialized.retention().toMillis() : windows.gracePeriodMs() + 2 * windows.timeDifferenceMs();
Review comment:
+1 to add a comment on the extra retention (here and in SlidingWindowedCoGroupedKStreamImpl)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
Review comment:
We need to check `leftWinAgg` for null, right? Also, is it ever possible for `leftWinAgg` to be non-null but not satisfy this condition? Maybe we can just check `leftWinAgg != null` and if so, then assert that `leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()` is always true (eg throw an `IllegalStateException` if it's not)
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ observedStreamTime = Math.max(observedStreamTime,
+ timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ //to keep find the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ //Fetch all the windows that have a start time <= timestamp and >= timestamp-2*timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long endTime = next.key.window().end();
+ final long startTime = next.key.window().start();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ continue;
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
Review comment:
Same here, let's not pin the start time to 0 for now and just drop the early records
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468214782
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
+ * records in the same window and given window grace period.
Review comment:
Haha, my specialty!
The distillation of this sentence is "Windows are defined based on a record's timestamp, window size, and window grace period." I think the meaning is pretty clear, so no need to change anything.
Just to point it out, there's structural ambiguity about whether the sentence is saying "a record's (timestamp, window size, window grace period)" (I.e., three properties of the record), or whether there are three top-level things that define the window. The latter was intended. I think actually inserting "the" before "window" both times would clear it up: "Windows are defined based on a record's timestamp, the window size, and the window grace period."
Another note is that because the second item in the list is so long, the structure of the list gets a little lost. It would be better in this case to use the Oxford comma to clearly delineate the boundary between the second and third items.
So, although I think this is fine as-is, if you want me to break out the red pen, I'd say:
```
* Sliding Windows are defined based on a record's timestamp, the window size based on the given maximum time difference (inclusive) between
* records in the same window, and the given window grace period.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465193735
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/SlidingWindowsTest.java
##########
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.junit.Test;
+
+import static java.time.Duration.ofMillis;
+import static org.apache.kafka.streams.EqualityCheck.verifyEquality;
+import static org.apache.kafka.streams.EqualityCheck.verifyInEquality;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.fail;
+
+@SuppressWarnings("deprecation")
+public class SlidingWindowsTest {
+
+ private static final long ANY_SIZE = 123L;
+
+ @Test
+ public void shouldSetWindowSize() {
+ assertEquals(ANY_SIZE, SlidingWindows.withTimeDifferenceAndGrace(ofMillis(ANY_SIZE), ofMillis(3)).timeDifferenceMs());
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeZero() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(0), ofMillis(5));
+ }
+
+ @Test(expected = IllegalArgumentException.class)
+ public void windowSizeMustNotBeNegative() {
+ SlidingWindows.withTimeDifferenceAndGrace(ofMillis(-1), ofMillis(5));
Review comment:
I think just confirming that the correct error will be thrown when someone sets a `timeDifference` we don't want. I'll update all the `windowSize` to be `timeDifference` and I agree, no need to check that it isn't 0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466425932
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
I'm happy to do a PR! Looking into it now though, `getStatefulProcessorNode` is called by `build`, so I think to really separate it by type we'd need a different `build` _and_ `StatefulProcessorNode`, otherwise we'd be moving the null checks into `build` and then calling the correct `getStatefulProcessorNode`, which does't seem to really fix anything. Thoughts? It's easy to create new `build` functions but I figured this might fall under not avoiding code duplication :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r474966465
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,579 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Random;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "2", 20L);
+ inputTopic1.pipeInput("A", "3", 22L);
+ inputTopic1.pipeInput("A", "4", 15L);
+
+ inputTopic1.pipeInput("B", "1", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "3", 18L);
+ inputTopic1.pipeInput("B", "4", 19L);
+ inputTopic1.pipeInput("B", "5", 25L);
+ inputTopic1.pipeInput("B", "6", 14L);
+
+ inputTopic1.pipeInput("C", "1", 11L);
+ inputTopic1.pipeInput("C", "2", 15L);
+ inputTopic1.pipeInput("C", "3", 16L);
+ inputTopic1.pipeInput("C", "4", 21);
+ inputTopic1.pipeInput("C", "5", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // FINAL WINDOW: A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2", 20),
+ // FINAL WINDOW: A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+3", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3", 22),
+ // FINAL WINDOW: A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+2+4", 20),
+ // FINAL WINDOW: A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+2+4", 20),
+ // FINAL WINDOW: A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+2+3+4", 22),
+ // FINAL WINDOW: A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+4", 15),
+ // FINAL WINDOW: A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+2+3", 22),
+
+ // FINAL WINDOW: B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+1", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // FINAL WINDOW: B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+1+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4", 19),
+ // FINAL WINDOW: B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+4+5", 25),
+ // FINAL WINDOW: B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+5", 25),
+ // FINAL WINDOW: B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+3+4+5", 25),
+ // FINAL WINDOW: B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+1+2+3+6", 18),
+ // FINAL WINDOW: B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+1+2+3+4+6", 19),
+ // FINAL WINDOW: B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+3+4+6", 19),
+ // FINAL WINDOW: B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+3+4+6", 19),
+ // FINAL WINDOW: B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+1+2+6", 14),
+
+ // FINAL WINDOW: C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+1", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2", 15),
+ // FINAL WINDOW: C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+1+2", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3", 16),
+ // FINAL WINDOW: C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+1+2+3", 16),
+ // FINAL WINDOW: C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+2+3+4", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4", 21),
+ // FINAL WINDOW: C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+1+2+3+4", 21),
+ // FINAL WINDOW: C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+3+4+5", 23),
+ // FINAL WINDOW: C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+4+5", 23),
+ // FINAL WINDOW: C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+5", 23),
+ // FINAL WINDOW: C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+2+3+4+5", 23),
+
+ // FINAL WINDOW: D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // FINAL WINDOW: D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // FINAL WINDOW: D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // FINAL WINDOW: D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // FINAL WINDOW: D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // FINAL WINDOW: D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+ final String topic2 = "topic2";
+
+ final KTable<Windowed<String>, String> table1 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table1.toStream().process(supplier);
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic2, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic2-Canonized").withValueSerde(Serdes.String())
+ );
+ table2.toStream().process(supplier);
+
+ table1.join(table2, (p1, p2) -> p1 + "%" + p2).toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ final TestInputTopic<String, String> inputTopic2 =
+ driver.createInputTopic(topic2, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("B", "2", 11L);
+ inputTopic1.pipeInput("C", "3", 12L);
+
+ final List<MockProcessor<Windowed<String>, String>> processors = supplier.capturedProcessors(3);
+
+ processors.get(0).checkAndClearProcessResult(
+ // left windows created by the first set of records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(1, 11)), "0+2", 11),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3", 12)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic1.pipeInput("A", "1", 15L);
+ inputTopic1.pipeInput("B", "2", 16L);
+ inputTopic1.pipeInput("C", "3", 19L);
+
+ processors.get(0).checkAndClearProcessResult(
+ // right windows from previous records are created, and left windows from new records to table 1
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(12, 22)), "0+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3", 19),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(9, 19)), "0+3+3", 19)
+ );
+ processors.get(1).checkAndClearProcessResult();
+ processors.get(2).checkAndClearProcessResult();
+
+ inputTopic2.pipeInput("A", "a", 10L);
+ inputTopic2.pipeInput("B", "b", 30L);
+ inputTopic2.pipeInput("C", "c", 12L);
+ inputTopic2.pipeInput("C", "c", 35L);
+
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // left windows from first set of records sent to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+b", 30),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+c", 12),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(25, 35)), "0+c", 35)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1%0+a", 10),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(2, 12)), "0+3%0+c", 12)
+ );
+
+ inputTopic2.pipeInput("A", "a", 15L);
+ inputTopic2.pipeInput("B", "b", 16L);
+ inputTopic2.pipeInput("C", "c", 17L);
+
+ processors.get(0).checkAndClearProcessResult();
+ processors.get(1).checkAndClearProcessResult(
+ // right windows from previous records are created (where applicable), and left windows from new records to table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+c", 17),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(7, 17)), "0+c+c", 17)
+ );
+ processors.get(2).checkAndClearProcessResult(
+ // set of join windows from windows created by table 1 and table 2
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1%0+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+1%0+a+a", 15),
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(6, 16)), "0+2+2%0+b", 16),
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+3%0+c", 19)
+ );
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingNullKey() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+ builder
+ .stream(topic, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(100)))
+ .aggregate(MockInitializer.STRING_INIT, MockAggregator.toStringInstance("+"), Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()));
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput(null, "1");
+ assertThat(appender.getMessages(), hasItem("Skipping record due to null key or value. value=[1] topic=[topic] partition=[0] offset=[0]"));
+ }
+ }
+
+ @Test
+ public void shouldLogAndMeterWhenSkippingExpiredWindowByGrace() {
+ final String builtInMetricsVersion = StreamsConfig.METRICS_LATEST;
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic = "topic";
+
+ final KStream<String, String> stream1 = builder.stream(topic, Consumed.with(Serdes.String(), Serdes.String()));
+ stream1.groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(90L)))
+ .aggregate(
+ () -> "",
+ MockAggregator.toStringInstance("+"),
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonicalized").withValueSerde(Serdes.String()).withCachingDisabled().withLoggingDisabled()
+ )
+ .toStream()
+ .map((key, value) -> new KeyValue<>(key.toString(), value))
+ .to("output");
+
+ props.setProperty(StreamsConfig.BUILT_IN_METRICS_VERSION_CONFIG, builtInMetricsVersion);
+
+ try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(KStreamSlidingWindowAggregate.class);
+ final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(topic, new StringSerializer(), new StringSerializer());
+ inputTopic.pipeInput("k", "100", 200L);
+ inputTopic.pipeInput("k", "0", 100L);
+ inputTopic.pipeInput("k", "1", 101L);
+ inputTopic.pipeInput("k", "2", 102L);
+ inputTopic.pipeInput("k", "3", 103L);
+ inputTopic.pipeInput("k", "4", 104L);
+ inputTopic.pipeInput("k", "5", 105L);
+ inputTopic.pipeInput("k", "6", 15L);
+
+ assertLatenessMetrics(driver, is(7.0), is(185.0), is(96.25));
+
+ assertThat(appender.getMessages(), hasItems(
+ // left window for k@100
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[1] timestamp=[100] window=[90,100] expiration=[110] streamTime=[200]",
+ // left window for k@101
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[2] timestamp=[101] window=[91,101] expiration=[110] streamTime=[200]",
+ // left window for k@102
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[3] timestamp=[102] window=[92,102] expiration=[110] streamTime=[200]",
+ // left window for k@103
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[4] timestamp=[103] window=[93,103] expiration=[110] streamTime=[200]",
+ // left window for k@104
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[5] timestamp=[104] window=[94,104] expiration=[110] streamTime=[200]",
+ // left window for k@105
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[6] timestamp=[105] window=[95,105] expiration=[110] streamTime=[200]",
+ // left window for k@15
+ "Skipping record for expired window. key=[k] topic=[topic] partition=[0] offset=[7] timestamp=[15] window=[5,15] expiration=[110] streamTime=[200]"
+ ));
+ final TestOutputTopic<String, String> outputTopic =
+ driver.createOutputTopic("output", new StringDeserializer(), new StringDeserializer());
+ assertThat(outputTopic.readRecord(), equalTo(new TestRecord<>("[k@190/200]", "+100", null, 200L)));
+ assertTrue(outputTopic.isEmpty());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testProcessorRandomInput() {
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(10000)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table.toStream().process(supplier);
+
+ final List<ValueAndTimestamp<String>> input = Arrays.asList(
+ ValueAndTimestamp.make("A", 10L),
+ ValueAndTimestamp.make("A", 15L),
+ ValueAndTimestamp.make("A", 16L),
+ ValueAndTimestamp.make("A", 18L),
+ ValueAndTimestamp.make("A", 30L),
+ ValueAndTimestamp.make("A", 40L),
+ ValueAndTimestamp.make("A", 55L),
+ ValueAndTimestamp.make("A", 56L),
+ ValueAndTimestamp.make("A", 58L),
+ ValueAndTimestamp.make("A", 58L),
+ ValueAndTimestamp.make("A", 62L),
+ ValueAndTimestamp.make("A", 63L),
+ ValueAndTimestamp.make("A", 63L),
+ ValueAndTimestamp.make("A", 63L),
+ ValueAndTimestamp.make("A", 76L),
+ ValueAndTimestamp.make("A", 77L),
+ ValueAndTimestamp.make("A", 80L)
+ );
+
+ final long seed = new Random().nextLong();
+ final Random shuffle = new Random(seed);
+ Collections.shuffle(input, shuffle);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ for (int i = 0; i < input.size(); i++) {
+ inputTopic1.pipeInput("A", input.get(i).value(), input.get(i).timestamp());
+ }
+ }
+
+ final Map<Long, ValueAndTimestamp<String>> results = new HashMap<>();
+
+ for (final KeyValueTimestamp<Object, Object> entry : supplier.theCapturedProcessor().processed) {
+ final Windowed<String> window = (Windowed<String>) entry.key();
+ final Long start = window.window().start();
+ final ValueAndTimestamp valueAndTimestamp = ValueAndTimestamp.make((String) entry.value(), entry.timestamp());
+ if (results.putIfAbsent(start, valueAndTimestamp) != null) {
+ results.replace(start, valueAndTimestamp);
+ }
+ }
+ randomEqualityCheck(results, seed);
+ }
+
+ private void randomEqualityCheck(final Map<Long, ValueAndTimestamp<String>> actual, final Long seed) {
+ final Map<Long, ValueAndTimestamp<String>> expected = new HashMap<>();
+ expected.put(0L, ValueAndTimestamp.make("0+A", 10L));
+ expected.put(11L, ValueAndTimestamp.make("0+A+A+A", 18L));
+ expected.put(5L, ValueAndTimestamp.make("0+A+A", 15L));
+ expected.put(16L, ValueAndTimestamp.make("0+A+A", 18L));
+ expected.put(6L, ValueAndTimestamp.make("0+A+A+A", 16L));
+ expected.put(17L, ValueAndTimestamp.make("0+A", 18L));
+ expected.put(8L, ValueAndTimestamp.make("0+A+A+A+A", 18L));
+ expected.put(20L, ValueAndTimestamp.make("0+A", 30L));
+ expected.put(31L, ValueAndTimestamp.make("0+A", 40L));
+ expected.put(30L, ValueAndTimestamp.make("0+A+A", 40L));
+ expected.put(45L, ValueAndTimestamp.make("0+A", 55L));
+ expected.put(56L, ValueAndTimestamp.make("0+A+A+A+A+A+A+A", 63L));
+ expected.put(46L, ValueAndTimestamp.make("0+A+A", 56L));
+ expected.put(57L, ValueAndTimestamp.make("0+A+A+A+A+A+A", 63L));
+ expected.put(48L, ValueAndTimestamp.make("0+A+A+A+A", 58L));
+ expected.put(59L, ValueAndTimestamp.make("0+A+A+A+A", 63L));
+ expected.put(52L, ValueAndTimestamp.make("0+A+A+A+A+A", 62L));
+ expected.put(63L, ValueAndTimestamp.make("0+A+A+A", 63L));
+ expected.put(53L, ValueAndTimestamp.make("0+A+A+A+A+A+A+A+A", 63L));
+ expected.put(66L, ValueAndTimestamp.make("0+A", 76L));
+ expected.put(77L, ValueAndTimestamp.make("0+A+A", 80L));
+ expected.put(67L, ValueAndTimestamp.make("0+A+A", 77L));
+ expected.put(78L, ValueAndTimestamp.make("0+A", 80L));
+ expected.put(70L, ValueAndTimestamp.make("0+A+A+A", 80L));
+
+ try {
+ assertEquals(expected, actual);
+ } catch (final AssertionError t) {
+ throw new AssertionError(
+ "Assertion failed in randomized test. Reproduce with seed: " + seed + ".",
+ t
+ );
+ } catch (final Throwable t) {
Review comment:
Can we actually wrap the whole `testProcessorRandomInput` test in the try-catch? Or at least, everything after the initial setup? Would be nice to have the seed in case something weird happens during the processing itself
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465382663
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/SlidingWindows.java
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream;
+
+import org.apache.kafka.streams.internals.ApiUtils;
+import org.apache.kafka.streams.processor.TimestampExtractor;
+import java.time.Duration;
+import java.util.Objects;
+import static org.apache.kafka.streams.internals.ApiUtils.prepareMillisCheckFailMsgPrefix;
+
+/**
+ /**
+ * A sliding window used for aggregating events.
+ * <p>
+ * Sliding Windows are defined based on a record's timestamp, window size based on the given maximum time difference (inclusive) between
Review comment:
It took me a second to understand the structure of this sentence, can we insert an `and` after the `record's timestamp`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468690429
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
Review comment:
I'm not sure if I'm just missing something, but it doesn't look like there's a way to check what retention is. I created a test to make sure anything lower than our bound throws an exception, but I can't find anywhere the retention time is exposed for me to check what it's set to
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r467296111
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
Clearly Kafka Streams is superior to the plain Consumer 😈
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463039985
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImpl.java
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.Reducer;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.internals.graph.StreamsGraphNode;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+
+import java.time.Duration;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.AGGREGATE_NAME;
+import static org.apache.kafka.streams.kstream.internals.KGroupedStreamImpl.REDUCE_NAME;
+
+public class SlidingWindowedKStreamImpl<K, V> extends AbstractStream<K, V> implements TimeWindowedKStream<K, V> {
+ private final SlidingWindows windows;
+ private final GroupedStreamAggregateBuilder<K, V> aggregateBuilder;
+
+ SlidingWindowedKStreamImpl(final SlidingWindows windows,
+ final InternalStreamsBuilder builder,
+ final Set<String> subTopologySourceNodes,
+ final String name,
+ final Serde<K> keySerde,
+ final Serde<V> valueSerde,
+ final GroupedStreamAggregateBuilder<K, V> aggregateBuilder,
+ final StreamsGraphNode streamsGraphNode) {
+ super(name, keySerde, valueSerde, subTopologySourceNodes, streamsGraphNode, builder);
+ this.windows = Objects.requireNonNull(windows, "windows can't be null");
+ this.aggregateBuilder = aggregateBuilder;
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count() {
+ return count(NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named) {
+ return doCount(named, Materialized.with(keySerde, Serdes.Long()));
+ }
+
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ return count(NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, Long> count(final Named named, final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ // TODO: remove this when we do a topology-incompatible release
+ // we used to burn a topology name here, so we have to keep doing it for compatibility
+ if (new MaterializedInternal<>(materialized).storeName() == null) {
+ builder.newStoreName(AGGREGATE_NAME);
+ }
+
+ return doCount(named, materialized);
+ }
+
+ private KTable<Windowed<K>, Long> doCount(final Named named,
+ final Materialized<K, Long, WindowStore<Bytes, byte[]>> materialized) {
+ final MaterializedInternal<K, Long, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(Serdes.Long());
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.countInitializer, aggregateBuilder.countAggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator) {
+ return aggregate(initializer, aggregator, Materialized.with(keySerde, null));
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named) {
+ return aggregate(initializer, aggregator, named, Materialized.with(keySerde, null));
+ }
+
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ return aggregate(initializer, aggregator, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public <VR> KTable<Windowed<K>, VR> aggregate(final Initializer<VR> initializer,
+ final Aggregator<? super K, ? super V, VR> aggregator,
+ final Named named,
+ final Materialized<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(initializer, "initializer can't be null");
+ Objects.requireNonNull(aggregator, "aggregator can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+ final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, AGGREGATE_NAME);
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+
+ final String aggregateName = new NamedInternal(named).orElseGenerateWithPrefix(builder, AGGREGATE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(aggregateName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), initializer, aggregator),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer) {
+ return reduce(reducer, NamedInternal.empty());
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer, final Named named) {
+ return reduce(reducer, named, Materialized.with(keySerde, valueSerde));
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ return reduce(reducer, NamedInternal.empty(), materialized);
+ }
+
+ @Override
+ public KTable<Windowed<K>, V> reduce(final Reducer<V> reducer,
+ final Named named,
+ final Materialized<K, V, WindowStore<Bytes, byte[]>> materialized) {
+ Objects.requireNonNull(reducer, "reducer can't be null");
+ Objects.requireNonNull(named, "named can't be null");
+ Objects.requireNonNull(materialized, "materialized can't be null");
+
+ final MaterializedInternal<K, V, WindowStore<Bytes, byte[]>> materializedInternal =
+ new MaterializedInternal<>(materialized, builder, REDUCE_NAME);
+
+ if (materializedInternal.keySerde() == null) {
+ materializedInternal.withKeySerde(keySerde);
+ }
+ if (materializedInternal.valueSerde() == null) {
+ materializedInternal.withValueSerde(valueSerde);
+ }
+
+ final String reduceName = new NamedInternal(named).orElseGenerateWithPrefix(builder, REDUCE_NAME);
+
+ return aggregateBuilder.build(
+ new NamedInternal(reduceName),
+ materialize(materializedInternal),
+ new KStreamSlidingWindowAggregate<>(windows, materializedInternal.storeName(), aggregateBuilder.reduceInitializer, aggregatorForReducer(reducer)),
+ materializedInternal.queryableStoreName(),
+ materializedInternal.keySerde() != null ? new FullTimeWindowedSerde<>(materializedInternal.keySerde(), windows.timeDifference()) : null,
+ materializedInternal.valueSerde());
+
+
+ }
+
+ @SuppressWarnings("deprecation") // continuing to support Windows#maintainMs/segmentInterval in fallback mode
+ private <VR> StoreBuilder<TimestampedWindowStore<K, VR>> materialize(final MaterializedInternal<K, VR, WindowStore<Bytes, byte[]>> materialized) {
+ WindowBytesStoreSupplier supplier = (WindowBytesStoreSupplier) materialized.storeSupplier();
+ if (supplier == null) {
+ // new style retention: use Materialized retention and default segmentInterval
+ final long retentionPeriod = materialized.retention().toMillis();
+
+ if ((windows.timeDifference() + windows.gracePeriodMs()) > retentionPeriod) {
Review comment:
That's a good point. I think that `>= 2* timeDifference + gracePeriod` makes sense. Adding gracePeriod is just to help with out-of-order records, right? Since for a normal record we won't need anything beyond 2*timeDifference
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r479362665
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWinAlreadyCreated && rightWindowIsNotEmpty(rightWinAgg, timestamp)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ private boolean rightWindowIsNotEmpty(final ValueAndTimestamp<Agg> rightWinAgg, final long timestamp) {
+ return rightWinAgg != null && rightWinAgg.timestamp() > timestamp;
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window,
+ final ValueAndTimestamp<Agg> valueAndTime,
+ final K key,
+ final V value,
+ final long closeTime,
+ final long timestamp) {
Review comment:
Yeah it's definitely vague, I'll update
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463047237
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
Review comment:
Yeah that's much clearer, and differentiates between the other bools better
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei merged pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei merged pull request #9039:
URL: https://github.com/apache/kafka/pull/9039
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#issuecomment-683917229
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465219522
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
Review comment:
To clarify, are you wanting to add records that would fall after the third batch _outside_ of all the existing windows, or so that it will fall into the third batch's windows but not the second batch's windows?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466042505
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
Review comment:
That should be covered in `KStreamSlidingWindowAggregateTest`, which goes through more of the edge cases using the `TopologyTestDriver` which is a little easier to manipulate than this set up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r471798256
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
Review comment:
```suggestion
final Set<Long> windowStartTimes = new HashSet<>();
```
Also I think this set is pretty clearly named, so we probably don't need a comment for it
##########
File path: checkstyle/suppressions.xml
##########
@@ -167,6 +167,9 @@
<suppress checks="(FinalLocalVariable|UnnecessaryParentheses|BooleanExpressionComplexity|CyclomaticComplexity|WhitespaceAfter|LocalVariableName)"
files="Murmur3.java"/>
+ <suppress checks="(CyclomaticComplexity)"
+ files="CogroupedStreamAggregateBuilder.java"/>
Review comment:
Do we still need this one after the cleanup you did?
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWinAlreadyCreated && rightWinIsNotEmpty(rightWinAgg, timestamp)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ private boolean rightWinIsNotEmpty(final ValueAndTimestamp<Agg> rightWinAgg, final long timestamp) {
Review comment:
nit: can we use the full word `Window` in method names at least
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "1", 20L);
+ inputTopic1.pipeInput("A", "1", 22L);
+ inputTopic1.pipeInput("A", "3", 15L);
+
+ inputTopic1.pipeInput("B", "2", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "2", 18L);
+ inputTopic1.pipeInput("B", "1", 19L);
+ inputTopic1.pipeInput("B", "2", 25L);
+ inputTopic1.pipeInput("B", "3", 14L);
+
+ inputTopic1.pipeInput("C", "3", 11L);
+ inputTopic1.pipeInput("C", "4", 15L);
+ inputTopic1.pipeInput("C", "1", 16L);
+ inputTopic1.pipeInput("C", "1", 21);
+ inputTopic1.pipeInput("C", "1", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // A@10 left window created when A@10 processed
Review comment:
For readability, could we mark the final results for each window? We want to make sure all the intermediate results are as expected, but what we really care about is what we got in the end. It would just help to have the critical output easier to find and get oriented in the tests
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L))))); }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER, Materialized.as("store")));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ null,
+ Materialized.as("store")));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, MockAggregator.TOSTRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null, Materialized.as("store")));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfNamedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Named) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnCountIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.count((Materialized<String, Long, WindowStore<Bytes, byte[]>>) null));
+ }
+
+ @Test
+ public void shouldThrowIllegalArgumentWhenRetentionIsTooSmall() {
+ assertThrows(IllegalArgumentException.class, () -> windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized
+ .<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withRetention(ofMillis(1L))
+ )
+ );
+ }
+
+ @Test
+ public void ShouldDropWindowsOutsideOfRetention() {
+ final WindowBytesStoreSupplier storeSupplies = Stores.inMemoryWindowStore("aggregated", ofMillis(1200L), ofMillis(100L), false);
Review comment:
```suggestion
final WindowBytesStoreSupplier storeSupplier = Stores.inMemoryWindowStore("aggregated", ofMillis(1200L), ofMillis(100L), false);
```
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L))))); }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER, Materialized.as("store")));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ null,
+ Materialized.as("store")));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, MockAggregator.TOSTRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null, Materialized.as("store")));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfNamedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Named) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnCountIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.count((Materialized<String, Long, WindowStore<Bytes, byte[]>>) null));
+ }
+
+ @Test
+ public void shouldThrowIllegalArgumentWhenRetentionIsTooSmall() {
+ assertThrows(IllegalArgumentException.class, () -> windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized
+ .<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withRetention(ofMillis(1L))
+ )
+ );
+ }
+
+ @Test
+ public void ShouldDropWindowsOutsideOfRetention() {
+ final WindowBytesStoreSupplier storeSupplies = Stores.inMemoryWindowStore("aggregated", ofMillis(1200L), ofMillis(100L), false);
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String>as(storeSupplies)
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withCachingDisabled());
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(TOPIC, new StringSerializer(), new StringSerializer());
+
+ inputTopic.pipeInput("1", "2", 100L);
+ inputTopic.pipeInput("1", "3", 500L);
+ inputTopic.pipeInput("1", "4", 1000L);
+ inputTopic.pipeInput("1", "5", 2000L);
+
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "1", ofEpochMilli(0), ofEpochMilli(10000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(900, 1000)), "0+4"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(1900, 2000)), "0+5"))));
Review comment:
nit: you could use the version of `fetch` that just takes a single key instead of a key range, since there's only one key here
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "1", 20L);
+ inputTopic1.pipeInput("A", "1", 22L);
+ inputTopic1.pipeInput("A", "3", 15L);
+
+ inputTopic1.pipeInput("B", "2", 12L);
Review comment:
It might be nice to use different values for each record (at least within the same key). I don't think there are really any edge cases we should worry about when records have the same value so we may as well use a distinct one to make the tests a bit easier to read
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final String topic1 = "topic1";
+
+ final KTable<Windowed<String>, String> table2 = builder
+ .stream(topic1, Consumed.with(Serdes.String(), Serdes.String()))
+ .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(10), ofMillis(50)))
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("topic1-Canonized").withValueSerde(Serdes.String())
+ );
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ table2.toStream().process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic1 =
+ driver.createInputTopic(topic1, new StringSerializer(), new StringSerializer());
+ inputTopic1.pipeInput("A", "1", 10L);
+ inputTopic1.pipeInput("A", "1", 20L);
+ inputTopic1.pipeInput("A", "1", 22L);
+ inputTopic1.pipeInput("A", "3", 15L);
+
+ inputTopic1.pipeInput("B", "2", 12L);
+ inputTopic1.pipeInput("B", "2", 13L);
+ inputTopic1.pipeInput("B", "2", 18L);
+ inputTopic1.pipeInput("B", "1", 19L);
+ inputTopic1.pipeInput("B", "2", 25L);
+ inputTopic1.pipeInput("B", "3", 14L);
+
+ inputTopic1.pipeInput("C", "3", 11L);
+ inputTopic1.pipeInput("C", "4", 15L);
+ inputTopic1.pipeInput("C", "1", 16L);
+ inputTopic1.pipeInput("C", "1", 21);
+ inputTopic1.pipeInput("C", "1", 23L);
+
+ inputTopic1.pipeInput("D", "4", 11L);
+ inputTopic1.pipeInput("D", "2", 12L);
+ inputTopic1.pipeInput("D", "3", 29L);
+ inputTopic1.pipeInput("D", "5", 16L);
+ }
+
+ assertEquals(
+ asList(
+ // A@10 left window created when A@10 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(0, 10)), "0+1", 10),
+ // A@10 right window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1", 20),
+ // A@20 left window created when A@20 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+1", 20),
+ // A@20 right window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(21, 31)), "0+1", 22),
+ // A@22 left window created when A@22 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+1+1", 22),
+ // A@20 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(10, 20)), "0+1+1+3", 20),
+ // A@10 right window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(11, 21)), "0+1+3", 20),
+ // A@22 left window updated when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(12, 22)), "0+1+1+3", 22),
+ // A@15 left window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(5, 15)), "0+1+3", 15),
+ // A@15 right window created when A@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(16, 26)), "0+1+1", 22),
+
+ // B@12 left window created when B@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(2, 12)), "0+2", 12),
+ // B@12 right window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2", 13),
+ // B@13 left window created when B@13 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(3, 13)), "0+2+2", 13),
+ // B@12 right window updated when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2", 18),
+ // B@13 right window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2", 18),
+ // B@18 left window created when B@18 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+2+2+2", 18),
+ // B@12 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2+1", 19),
+ // B@13 right window updated when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2+1", 19),
+ // B@18 right window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+1", 19),
+ // B@19 left window created when B@19 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+2+2+2+1", 19),
+ // B@18 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(19, 29)), "0+1+2", 25),
+ // B@19 right window updated when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(20, 30)), "0+2", 25),
+ // B@25 left window created when B@25 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(15, 25)), "0+2+1+2", 25),
+ // B@18 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(8, 18)), "0+2+2+2+3", 18),
+ // B@19 left window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(9, 19)), "0+2+2+2+1+3", 19),
+ // B@12 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(13, 23)), "0+2+2+1+3", 19),
+ // B@13 right window updated when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(14, 24)), "0+2+1+3", 19),
+ // B@14 left window created when B@14 processed
+ new KeyValueTimestamp<>(new Windowed<>("B", new TimeWindow(4, 14)), "0+2+2+3", 14),
+
+ // C@11 left window created when C@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(1, 11)), "0+3", 11),
+ // C@11 right window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4", 15),
+ // C@15 left window created when C@15 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(5, 15)), "0+3+4", 15),
+ // C@11 right window updated when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4+1", 16),
+ // C@15 right window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1", 16),
+ // C@16 left window created when C@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(6, 16)), "0+3+4+1", 16),
+ // C@11 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(12, 22)), "0+4+1+1", 21),
+ // C@15 right window updated when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1+1", 21),
+ // C@16 right window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+1", 21),
+ // C@21 left window created when C@21 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(11, 21)), "0+3+4+1+1", 21),
+ // C@15 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(16, 26)), "0+1+1+1", 23),
+ // C@16 right window updated when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(17, 27)), "0+1+1", 23),
+ // C@21 right window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(22, 32)), "0+1", 23),
+ // C@23 left window created when C@23 processed
+ new KeyValueTimestamp<>(new Windowed<>("C", new TimeWindow(13, 23)), "0+4+1+1+1", 23),
+
+ // D@11 left window created when D@11 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(1, 11)), "0+4", 11),
+ // D@11 right window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2", 12),
+ // D@12 left window created when D@12 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(2, 12)), "0+4+2", 12),
+ // D@29 left window created when D@29 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(19, 29)), "0+3", 29),
+ // D@11 right window updated when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(12, 22)), "0+2+5", 16),
+ // D@12 right window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(13, 23)), "0+5", 16),
+ // D@16 left window created when D@16 processed
+ new KeyValueTimestamp<>(new Windowed<>("D", new TimeWindow(6, 16)), "0+4+2+5", 16)
+ ),
+ supplier.theCapturedProcessor().processed
+ );
+ }
+
+ @Test
+ public void testJoin() {
Review comment:
I still don't exactly understand why we have a join test in the `KStreamXXWindowAggregateTest`, but thanks for adding it for sliding windows. I'm sure there was a good reason for it, probably long ago
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregateTest.java
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.MetricName;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.KeyValueTimestamp;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TestOutputTopic;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.KTable;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.processor.internals.testutil.LogCaptureAppender;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.streams.test.TestRecord;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessor;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.hamcrest.Matcher;
+import org.junit.Test;
+
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.util.Arrays.asList;
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.hasItems;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.CoreMatchers.not;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+public class KStreamSlidingWindowAggregateTest {
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private final String threadId = Thread.currentThread().getName();
+
+ @Test
+ public void testAggBasic() {
Review comment:
Sorry that I only just got to looking through this class 😞 . The tests here look good but can we add some more test coverage of possible edge cases? I know we can't test early records until the next PR, but we should probably have more than just the one test of the core functionality.
I know it's really annoying to have to think through all the intermediate output, so maybe you can write a helper method that just grabs the final result of each window in the output? Then we could have a number of tests that go through a larger number of input records without you having to spend all day manually processing them yourself 😄
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L))))); }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER, Materialized.as("store")));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ null,
+ Materialized.as("store")));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, MockAggregator.TOSTRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null, Materialized.as("store")));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfNamedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Named) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnCountIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.count((Materialized<String, Long, WindowStore<Bytes, byte[]>>) null));
+ }
+
+ @Test
+ public void shouldThrowIllegalArgumentWhenRetentionIsTooSmall() {
+ assertThrows(IllegalArgumentException.class, () -> windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized
+ .<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withRetention(ofMillis(1L))
+ )
+ );
+ }
+
+ @Test
+ public void ShouldDropWindowsOutsideOfRetention() {
Review comment:
```suggestion
public void shouldDropWindowsOutsideOfRetention() {
```
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L))))); }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER, Materialized.as("store")));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ null,
+ Materialized.as("store")));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, MockAggregator.TOSTRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null, Materialized.as("store")));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfNamedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Named) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnCountIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.count((Materialized<String, Long, WindowStore<Bytes, byte[]>>) null));
+ }
+
+ @Test
+ public void shouldThrowIllegalArgumentWhenRetentionIsTooSmall() {
+ assertThrows(IllegalArgumentException.class, () -> windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized
+ .<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withRetention(ofMillis(1L))
+ )
+ );
+ }
+
+ @Test
+ public void ShouldDropWindowsOutsideOfRetention() {
+ final WindowBytesStoreSupplier storeSupplies = Stores.inMemoryWindowStore("aggregated", ofMillis(1200L), ofMillis(100L), false);
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String>as(storeSupplies)
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withCachingDisabled());
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(TOPIC, new StringSerializer(), new StringSerializer());
+
+ inputTopic.pipeInput("1", "2", 100L);
+ inputTopic.pipeInput("1", "3", 500L);
+ inputTopic.pipeInput("1", "4", 1000L);
Review comment:
Can we insert one that's like right on the border of the retention period? So if the streamtime at the end is 2,000 then the window cut off is 800 (or start time of 700), and verify that anything starting before 699 is gone and everything after that is there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#issuecomment-684074223
Since Jenkins PR builds are still not functioning, I've merged in trunk and verified this pull request locally before merging it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463036048
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
Review comment:
done!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478716769
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ //left window just contains the current record
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(Math.max(0, timestamp - windows.timeDifferenceMs()), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create the right window for the current record, if need be
+ if (!rightWinAlreadyCreated && rightWinAgg != null && rightWinAgg.timestamp() > timestamp) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ public void processInOrder(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
Review comment:
To be honest, it might not be so bad to just leave things as is and drop early records, since any sensible timestamps are unlikely to be that close to the epoch. But I do believe users may want to use lower timestamps in their unit testing (`1598565116374` is not a very human readable number) and would be surprised to see these records just dropped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r468796354
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
Review comment:
Update: the windows themselves are the same but the value is different for each test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r472441802
##########
File path: streams/src/test/java/org/apache/kafka/streams/kstream/internals/SlidingWindowedKStreamImplTest.java
##########
@@ -0,0 +1,438 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Grouped;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.Named;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.kstream.TimeWindowedKStream;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.TestInputTopic;
+import org.apache.kafka.test.MockAggregator;
+import org.apache.kafka.test.MockInitializer;
+import org.apache.kafka.test.MockProcessorSupplier;
+import org.apache.kafka.test.MockReducer;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+
+import static java.time.Duration.ofMillis;
+import static java.time.Instant.ofEpochMilli;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+
+public class SlidingWindowedKStreamImplTest {
+
+ private static final String TOPIC = "input";
+ private final StreamsBuilder builder = new StreamsBuilder();
+ private final Properties props = StreamsTestUtils.getStreamsConfig(Serdes.String(), Serdes.String());
+ private TimeWindowedKStream<String, String> windowedStream;
+
+ @Before
+ public void before() {
+ final KStream<String, String> stream = builder.stream(TOPIC, Consumed.with(Serdes.String(), Serdes.String()));
+ windowedStream = stream.
+ groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(100L), ofMillis(1000L)));
+ }
+
+ @Test
+ public void shouldCountSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, Long> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .count()
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make(1L, 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(2L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make(1L, 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make(2L, 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make(1L, 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make(1L, 200L)));
+ }
+
+ @Test
+ public void shouldReduceSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .reduce(MockReducer.STRING_ADDER)
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("10", 200L)));
+ }
+
+ @Test
+ public void shouldAggregateSlidingWindows() {
+ final MockProcessorSupplier<Windowed<String>, String> supplier = new MockProcessorSupplier<>();
+ windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.with(Serdes.String(), Serdes.String()))
+ .toStream()
+ .process(supplier);
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ }
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(0L, 100L))),
+ equalTo(ValueAndTimestamp.make("0+1", 100L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(101L, 201L))),
+ equalTo(ValueAndTimestamp.make("0+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+1+2", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("1", new TimeWindow(400L, 500L))),
+ equalTo(ValueAndTimestamp.make("0+3", 500L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(100L, 200L))),
+ equalTo(ValueAndTimestamp.make("0+10+20", 200L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(50L, 150L))),
+ equalTo(ValueAndTimestamp.make("0+20", 150L)));
+ assertThat(
+ supplier.theCapturedProcessor().lastValueAndTimestampPerKey
+ .get(new Windowed<>("2", new TimeWindow(151L, 251L))),
+ equalTo(ValueAndTimestamp.make("0+10", 200L)));
+ }
+
+ @Test
+ public void shouldMaterializeCount() {
+ windowedStream.count(
+ Materialized.<String, Long, WindowStore<Bytes, byte[]>>as("count-store")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.Long()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, Long> windowStore = driver.getWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, Long>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), 2L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), 1L),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), 1L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), 2L),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), 1L))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("count-store");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make(1L, 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make(2L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make(1L, 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make(1L, 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make(2L, 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make(1L, 200L))))); }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeReduced() {
+ windowedStream.reduce(
+ MockReducer.STRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("reduced")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("reduced");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldMaterializeAggregated() {
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String()));
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ processData(driver);
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), "0+1"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), "0+1+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), "0+2"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), "0+3"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), "0+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), "0+10+20"),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), "0+10"))));
+ }
+ {
+ final WindowStore<String, ValueAndTimestamp<Long>> windowStore =
+ driver.getTimestampedWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, ValueAndTimestamp<Long>>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "2", ofEpochMilli(0), ofEpochMilli(1000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(0, 100)), ValueAndTimestamp.make("0+1", 100L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+1+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(101, 201)), ValueAndTimestamp.make("0+2", 150L)),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(400, 500)), ValueAndTimestamp.make("0+3", 500L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(50, 150)), ValueAndTimestamp.make("0+20", 150L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(100, 200)), ValueAndTimestamp.make("0+10+20", 200L)),
+ KeyValue.pair(new Windowed<>("2", new TimeWindow(151, 251)), ValueAndTimestamp.make("0+10", 200L)))));
+ }
+ }
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfInitializerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(null, MockAggregator.TOSTRING_ADDER, Materialized.as("store")));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfAggregatorIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ null,
+ Materialized.as("store")));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldThrowNullPointerOnMaterializedAggregateIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.aggregate(MockInitializer.STRING_INIT, MockAggregator.TOSTRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnMaterializedReduceIfReducerIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(null, Materialized.as("store")));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Materialized) null));
+ }
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void shouldThrowNullPointerOnMaterializedReduceIfNamedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.reduce(MockReducer.STRING_ADDER, (Named) null));
+ }
+
+ @Test
+ public void shouldThrowNullPointerOnCountIfMaterializedIsNull() {
+ assertThrows(NullPointerException.class, () -> windowedStream.count((Materialized<String, Long, WindowStore<Bytes, byte[]>>) null));
+ }
+
+ @Test
+ public void shouldThrowIllegalArgumentWhenRetentionIsTooSmall() {
+ assertThrows(IllegalArgumentException.class, () -> windowedStream
+ .aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized
+ .<String, String, WindowStore<Bytes, byte[]>>as("aggregated")
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withRetention(ofMillis(1L))
+ )
+ );
+ }
+
+ @Test
+ public void ShouldDropWindowsOutsideOfRetention() {
+ final WindowBytesStoreSupplier storeSupplies = Stores.inMemoryWindowStore("aggregated", ofMillis(1200L), ofMillis(100L), false);
+ windowedStream.aggregate(
+ MockInitializer.STRING_INIT,
+ MockAggregator.TOSTRING_ADDER,
+ Materialized.<String, String>as(storeSupplies)
+ .withKeySerde(Serdes.String())
+ .withValueSerde(Serdes.String())
+ .withCachingDisabled());
+
+ try (final TopologyTestDriver driver = new TopologyTestDriver(builder.build(), props)) {
+ final TestInputTopic<String, String> inputTopic =
+ driver.createInputTopic(TOPIC, new StringSerializer(), new StringSerializer());
+
+ inputTopic.pipeInput("1", "2", 100L);
+ inputTopic.pipeInput("1", "3", 500L);
+ inputTopic.pipeInput("1", "4", 1000L);
+ inputTopic.pipeInput("1", "5", 2000L);
+
+ {
+ final WindowStore<String, String> windowStore = driver.getWindowStore("aggregated");
+ final List<KeyValue<Windowed<String>, String>> data =
+ StreamsTestUtils.toList(windowStore.fetch("1", "1", ofEpochMilli(0), ofEpochMilli(10000L)));
+ assertThat(data, equalTo(Arrays.asList(
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(900, 1000)), "0+4"),
+ KeyValue.pair(new Windowed<>("1", new TimeWindow(1900, 2000)), "0+5"))));
Review comment:
Oh right, forgot that it doesn't have the window times either. Nevermind then
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478756346
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ //create left window for new record
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //there's a right window that the new record could create --> new record's left window is not empty
+ if (latestLeftTypeWindow != null) {
+ valueAndTime = ValueAndTimestamp.make(leftWinAgg.value(), timestamp);
+ } else {
+ valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ }
+ final TimeWindow window = new TimeWindow(timestamp - windows.timeDifferenceMs(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ //create right window for new record
+ if (!rightWinAlreadyCreated && rightWindowIsNotEmpty(rightWinAgg, timestamp)) {
+ final TimeWindow window = new TimeWindow(timestamp + 1, timestamp + 1 + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(getValueOrNull(rightWinAgg), Math.max(rightWinAgg.timestamp(), timestamp));
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ }
+
+ private boolean rightWindowIsNotEmpty(final ValueAndTimestamp<Agg> rightWinAgg, final long timestamp) {
+ return rightWinAgg != null && rightWinAgg.timestamp() > timestamp;
+ }
+
+ private boolean isLeftWindow(final KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> window) {
+ return window.key.window().end() == window.value.timestamp();
+ }
+
+ private void putAndForward(final Window window,
+ final ValueAndTimestamp<Agg> valueAndTime,
+ final K key,
+ final V value,
+ final long closeTime,
+ final long timestamp) {
Review comment:
Now that you bring it up, that's kind of a weird case for this method, and it's currently handled in a pretty subtle way. For example down on line 233 we are effectively checking for this case, and line 234 just happens to work correctly for it. But it's not at all obvious that we're even handling this case. Can we avoid the ternary operator when setting `newAgg` and `newTimestamp` and just use a normal if/else to explicitly set both of these for the special case? (ie `if (windowStart == timestamp + 1)`...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465371635
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
+ * windowed aggregations.
+ *
+ * @param windows the specification of the aggregation {@link SlidingWindows}
+ * @return an instance of {@link TimeWindowedCogroupedKStream}
+ */
+ TimeWindowedCogroupedKStream<K, VOut> windowedBy(final SlidingWindows windows);
Review comment:
Seems like the cogroup stuff makes up a pretty small amount of the overall PR, but up to Leah
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466032804
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
Review comment:
Thanks for confirming. Let keep this issue out for this PR than.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465334577
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/CogroupedKStream.java
##########
@@ -275,6 +275,15 @@
*/
<W extends Window> TimeWindowedCogroupedKStream<K, VOut> windowedBy(final Windows<W> windows);
+ /**
+ * Create a new {@link TimeWindowedCogroupedKStream} instance that can be used to perform sliding
+ * windowed aggregations.
+ *
+ * @param windows the specification of the aggregation {@link SlidingWindows}
+ * @return an instance of {@link TimeWindowedCogroupedKStream}
+ */
+ TimeWindowedCogroupedKStream<K, VOut> windowedBy(final SlidingWindows windows);
Review comment:
We could, but it would only pull out 3ish classes and not very many lines, so I don't think it would make this PR feel much smaller
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478728829
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+ final long startTime = next.key.window().start();
+ final long endTime = startTime + windows.timeDifferenceMs();
+
+ if (endTime < timestamp) {
+ leftWinAgg = next.value;
+ if (isLeftWindow(next)) {
+ latestLeftTypeWindow = next.key.window();
+ }
+ } else if (endTime == timestamp) {
+ leftWinAlreadyCreated = true;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else if (endTime > timestamp && startTime <= timestamp) {
+ rightWinAgg = next.value;
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ } else {
+ rightWinAlreadyCreated = true;
+ }
+ }
+ }
+
+ //create right window for previous record
+ if (latestLeftTypeWindow != null) {
+ final long rightWinStart = latestLeftTypeWindow.end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
Review comment:
While the range might give a window that starts and ends before the current record's timestamp, the current record would fall into the right window of the records _within_ those windows.
EX: TimeDifference = 10, record @ 30, range from (10,31). The earliest start time of a window we can have is 10, so the earliest `leftTypeWindow` we can find is from [10,20]. If there's a record at 2, it's right window would be [21,31], which our record @ 30 would fall within. Because this is true for the furthest possible record, it'll be true for the others that we find.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463056420
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
+ boolean rightWindowExists = false;
+ //to determine if we're creating the previous record's right window, helps with determining empty windows
+ boolean prevRightWinExists = false;
+ boolean prevRightWinAlreadyCreated = false;
+
+ try (
+ //Fetch all the windows that have a start time between timestamp and timestamp+timeDifference
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(key,
+ key,
+ Instant.ofEpochMilli(timestamp - 2 * windows.timeDifference()),
+ Instant.ofEpochMilli(timestamp + 1))
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundFirst = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundFirstEndTime = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists
+ if (next.key.window().start() == timestamp + 1) {
+ rightWindowExists = true;
+ continue;
Review comment:
I'm not sure if it can be moved out of the loop unless we also move a check for if the first is a window whose aggregate we need to update, which is easy to do but gets back to the redundant code that the algorithm had before having one while loop. I did update the comment though
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465097558
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/CogroupedStreamAggregateBuilder.java
##########
@@ -132,16 +135,19 @@
final boolean stateCreated,
final StoreBuilder<?> storeBuilder,
final Windows<W> windows,
+ final SlidingWindows slidingWindows,
final SessionWindows sessionWindows,
final Merger<? super K, VOut> sessionMerger) {
final ProcessorSupplier<K, ?> kStreamAggregate;
- if (windows == null && sessionWindows == null) {
+ if (windows == null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamAggregate<>(storeBuilder.name(), initializer, aggregator);
- } else if (windows != null && sessionWindows == null) {
+ } else if (windows != null && slidingWindows == null && sessionWindows == null) {
kStreamAggregate = new KStreamWindowAggregate<>(windows, storeBuilder.name(), initializer, aggregator);
- } else if (windows == null && sessionMerger != null) {
+ } else if (windows == null && slidingWindows != null && sessionWindows == null) {
+ kStreamAggregate = new KStreamSlidingWindowAggregate<>(slidingWindows, storeBuilder.name(), initializer, aggregator);
+ } else if (windows == null && slidingWindows == null && sessionMerger != null) {
Review comment:
Well, yes and no. We can follow two strategies: (1) we rely on the user to only set all `null` (for the non-windowed aggregation case) or either one of `windows`, `slidingWindow`, or `sessionWindow+sessionMerger` and we don't do any verification if the method is called correctly or not. However, for this case, we don't need to do any redundant not-null check and we could just write:
```
if (windows != null) {
...
} else if (slidingWindow != null) {
...
} else if (sessionWindow != null) { // we blindly assume that `sessionMerger` is not-null, too, for this case
...
} else { // all `null`: non-winowed aggregation
...
}
```
Or, (2) we do not "trust" the caller and do a proper check that the provided arguments make sense. And the existing code already has such a safe guard and does checks that not multiple windows are passed in and throws an `IllegalArgumentException` if the caller makes a mistake. I personally prefer to have a safe guard (especially on the non-hot code path) as it may prevent bugs. However, the current check is not complete, as it does not verify that `sessionMerger` must be not-null when `sessionWindows` is not-null; this may lead to a potentially cryptic `NullPointerException` later that is harder to understand. If we do the check and throw a proper `IllegalArgumentException("sessionMerger cannot be null for sessionWindows")` and `IllegalArgumentException("Unexpected sessionMerger parameter: should be null because sessionWindows is null");` help to identify the issue quickly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r463046306
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Instant;
+import java.util.HashSet;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ // flag to help determine if out-of-order record’s right window is non-empty
+ boolean foundLeftFirst = false;
+ //if current record's left/right windows already exist
+ boolean leftWinExists = false;
Review comment:
Yeah this makes sense, the boolean naming has definitely been a struggle. I think it's clearer actually if I change `prevRightWinExists` to `prevRightWinPossible` since that's what we're saying. I agree that for `leftWinExists` and `rightWinExists`, `alreadyCreated` makes more sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465191505
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
There seems to be a bug in `TimeWindowedDeserializer` related to [this ticket](https://issues.apache.org/jira/browse/KAFKA-4468) that ends up setting the windowSize to `Long.MAX_VALUE`. For the purposes of testing, I don't think having it as the max value is totally awful (just somewhat awful) and the window end calculations are all tested in a different set of tests done through topology driver. I'll make a ticket for this bug and try to get it fixed when I'm done with testing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465380753
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
Review comment:
Just to clarify, we don't get any additional output when the stream time is advanced and older windows drop out of the grace period. We've already forwarded their final state when the last record to update that window was processed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466052825
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,391 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ public void processReverse(final K key, final V value) {
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+ //store start times of windows we find
+ final HashSet<Long> windowStartTimes = new HashSet<Long>();
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
+ //if we've already seen the window with the closest start time to the record
+ boolean foundRightWinAgg = false;
+ //if we've already seen the window with the closest end time to the record
+ boolean foundLeftWinAgg = false;
+ while (iterator.hasNext()) {
+ next = iterator.next();
+ windowStartTimes.add(next.key.window().start());
+
+ //determine if current record's right window exists, will only be true at most once, on the first pass
+ if (next.key.window().start() == timestamp + 1) {
+ rightWinAlreadyCreated = true;
+ continue;
+ } else if (next.key.window().end() > timestamp) {
+ if (!foundRightWinAgg) {
+ foundRightWinAgg = true;
+ rightWinAgg = next.value;
+ }
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ continue;
+ } else if (next.key.window().end() == timestamp) {
+ putAndForward(next.key.window(), next.value, key, value, closeTime, timestamp);
+ leftWinAlreadyCreated = true;
+ continue;
+ } else {
+ if (!foundLeftWinAgg) {
+ leftWinAgg = next.value;
+ foundLeftWinAgg = true;
+ }
+ //If it's a left window, there is a record at this window's end time who may need a corresponding right window
+ if (isLeftWindow(next)) {
+ final long rightWinStart = next.key.window().end() + 1;
+ if (!windowStartTimes.contains(rightWinStart)) {
+ final TimeWindow window = new TimeWindow(rightWinStart, rightWinStart + windows.timeDifferenceMs());
+ final ValueAndTimestamp<Agg> valueAndTime = ValueAndTimestamp.make(initializer.apply(), timestamp);
+ putAndForward(window, valueAndTime, key, value, closeTime, timestamp);
+ }
+ break;
+ }
+ }
+ }
+ }
+ //create the left window of the current record if it's not created
+ if (!leftWinAlreadyCreated) {
+ final ValueAndTimestamp<Agg> valueAndTime;
+ //confirms that the left window contains more than the current record
+ if (leftWinAgg.timestamp() < timestamp && leftWinAgg.timestamp() > timestamp - windows.timeDifferenceMs()) {
Review comment:
Sounds good. We definitely need the null check just to avoid getting an NPE, but whether we _only_ need the null check is something we should put to the test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465374004
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
Review comment:
We definitely have the issue in all processors right now lol. It's not any more of a problem for this sliding windows algorithm as for any other operator that defines a grace period, at least. We might end up not dropping a late record that we should have; for sliding windows we'd get one extra window (with this record at the window end) whereas for a hopping/tumbling window we'd get N extra windows (however many overlaps there are)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465374004
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.time.Instant;
+import java.util.HashSet;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+ private boolean reverseIteratorImplemented = false;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ //catch unsupported operation error
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (reverseIteratorImplemented) {
+ processReverse(key, value);
+ } else {
+ processInOrder(key, value);
+ }
+ }
+
+ @SuppressWarnings("checkstyle:LineLength")
+ public void processReverse(final K key, final V value) {
+ if (key == null) {
+ log.warn(
+ "Skipping record due to null key. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
Review comment:
We definitely have the issue in all processors right now lol. It's not any more of a problem for this sliding windows algorithm as for any other operator that defines a grace period, at least. We might end up not dropping a late record that we should have; for sliding windows we'd get one extra window (with this record at the window end) whereas for a hopping/tumbling window we'd get N extra windows
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r465205088
##########
File path: streams/src/test/java/org/apache/kafka/streams/processor/internals/GlobalStreamThreadTest.java
##########
@@ -221,7 +221,6 @@ public void shouldTransitionToRunningOnStart() throws Exception {
globalStreamThread.shutdown();
}
- @Test
Review comment:
Whoops, not on purpose. Thanks for the check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#issuecomment-668325203
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466026007
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
+
+ final Serde<Windowed<String>> windowedSerde = WindowedSerdes.timeWindowedSerdeFrom(String.class);
+ groupedStream
+ .windowedBy(SlidingWindows.withTimeDifferenceAndGrace(ofMillis(timeDifference), ofMillis(2000L)))
+ .reduce(reducer)
+ .toStream()
+ .to(outputTopic, Produced.with(windowedSerde, Serdes.String()));
+
+ startStreams();
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> windowedOutput = receiveMessages(
+ new TimeWindowedDeserializer<>(),
+ new StringDeserializer(),
+ String.class,
+ 25);
+
+ // read from ConsoleConsumer
+ final String resultFromConsoleConsumer = readWindowedKeyedMessagesViaConsoleConsumer(
+ new TimeWindowedDeserializer<String>(),
+ new StringDeserializer(),
+ String.class,
+ 25,
+ true);
+
+ final Comparator<KeyValueTimestamp<Windowed<String>, String>> comparator =
+ Comparator.comparing((KeyValueTimestamp<Windowed<String>, String> o) -> o.key().key())
+ .thenComparing(KeyValueTimestamp::value);
+
+ windowedOutput.sort(comparator);
+ final long firstBatchLeftWindow = firstBatchTimestamp - timeDifference;
+ final long firstBatchRightWindow = firstBatchTimestamp + 1;
+ final long secondBatchLeftWindow = secondBatchTimestamp - timeDifference;
+ final long secondBatchRightWindow = secondBatchTimestamp + 1;
+ final long thirdBatchLeftWindow = thirdBatchTimestamp - timeDifference;
+
+ final List<KeyValueTimestamp<Windowed<String>, String>> expectResult = Arrays.asList(
+ new KeyValueTimestamp<>(new Windowed<>("A", new TimeWindow(firstBatchLeftWindow, Long.MAX_VALUE)), "A", firstBatchTimestamp),
Review comment:
The ticket is already fixed. You need to pass in the `windowSize` into the the constructor of `TimeWindowDeserializer` to get rid of the problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] lct45 commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
lct45 commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r478753568
##########
File path: streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSlidingWindowAggregate.java
##########
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.kstream.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.metrics.Sensor;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.kstream.Aggregator;
+import org.apache.kafka.streams.kstream.Initializer;
+import org.apache.kafka.streams.kstream.Window;
+import org.apache.kafka.streams.kstream.Windowed;
+import org.apache.kafka.streams.kstream.SlidingWindows;
+import org.apache.kafka.streams.processor.AbstractProcessor;
+import org.apache.kafka.streams.processor.Processor;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.internals.InternalProcessorContext;
+import org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.TimestampedWindowStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import java.util.HashSet;
+import java.util.Set;
+
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrLateRecordDropSensor;
+import static org.apache.kafka.streams.processor.internals.metrics.TaskMetrics.droppedRecordsSensorOrSkippedRecordsSensor;
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+public class KStreamSlidingWindowAggregate<K, V, Agg> implements KStreamAggProcessorSupplier<K, Windowed<K>, V, Agg> {
+ private final Logger log = LoggerFactory.getLogger(getClass());
+
+ private final String storeName;
+ private final SlidingWindows windows;
+ private final Initializer<Agg> initializer;
+ private final Aggregator<? super K, ? super V, Agg> aggregator;
+
+ private boolean sendOldValues = false;
+
+ public KStreamSlidingWindowAggregate(final SlidingWindows windows,
+ final String storeName,
+ final Initializer<Agg> initializer,
+ final Aggregator<? super K, ? super V, Agg> aggregator) {
+ this.windows = windows;
+ this.storeName = storeName;
+ this.initializer = initializer;
+ this.aggregator = aggregator;
+ }
+
+ @Override
+ public Processor<K, V> get() {
+ return new KStreamSlidingWindowAggregateProcessor();
+ }
+
+ public SlidingWindows windows() {
+ return windows;
+ }
+
+ @Override
+ public void enableSendingOldValues() {
+ sendOldValues = true;
+ }
+
+ private class KStreamSlidingWindowAggregateProcessor extends AbstractProcessor<K, V> {
+ private TimestampedWindowStore<K, Agg> windowStore;
+ private TimestampedTupleForwarder<Windowed<K>, Agg> tupleForwarder;
+ private StreamsMetricsImpl metrics;
+ private InternalProcessorContext internalProcessorContext;
+ private Sensor lateRecordDropSensor;
+ private Sensor droppedRecordsSensor;
+ private long observedStreamTime = ConsumerRecord.NO_TIMESTAMP;
+
+ @SuppressWarnings("unchecked")
+ @Override
+ public void init(final ProcessorContext context) {
+ super.init(context);
+ internalProcessorContext = (InternalProcessorContext) context;
+ metrics = internalProcessorContext.metrics();
+ final String threadId = Thread.currentThread().getName();
+ lateRecordDropSensor = droppedRecordsSensorOrLateRecordDropSensor(
+ threadId,
+ context.taskId().toString(),
+ internalProcessorContext.currentNode().name(),
+ metrics
+ );
+ droppedRecordsSensor = droppedRecordsSensorOrSkippedRecordsSensor(threadId, context.taskId().toString(), metrics);
+ windowStore = (TimestampedWindowStore<K, Agg>) context.getStateStore(storeName);
+ tupleForwarder = new TimestampedTupleForwarder<>(
+ windowStore,
+ context,
+ new TimestampedCacheFlushListener<>(context),
+ sendOldValues);
+ }
+
+ @Override
+ public void process(final K key, final V value) {
+ if (key == null || value == null) {
+ log.warn(
+ "Skipping record due to null key or value. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+
+ final long timestamp = context().timestamp();
+ //don't process records that don't fall within a full sliding window
+ if (timestamp < windows.timeDifferenceMs()) {
+ log.warn(
+ "Skipping record due to early arrival. value=[{}] topic=[{}] partition=[{}] offset=[{}]",
+ value, context().topic(), context().partition(), context().offset()
+ );
+ droppedRecordsSensor.record();
+ return;
+ }
+ processInOrder(key, value, timestamp);
+ }
+
+ public void processInOrder(final K key, final V value, final long timestamp) {
+
+ observedStreamTime = Math.max(observedStreamTime, timestamp);
+ final long closeTime = observedStreamTime - windows.gracePeriodMs();
+
+ //store start times of windows we find
+ final Set<Long> windowStartTimes = new HashSet<>();
+
+ // aggregate that will go in the current record’s left/right window (if needed)
+ ValueAndTimestamp<Agg> leftWinAgg = null;
+ ValueAndTimestamp<Agg> rightWinAgg = null;
+
+ //if current record's left/right windows already exist
+ boolean leftWinAlreadyCreated = false;
+ boolean rightWinAlreadyCreated = false;
+
+ // keep the left type window closest to the record
+ Window latestLeftTypeWindow = null;
+ try (
+ final KeyValueIterator<Windowed<K>, ValueAndTimestamp<Agg>> iterator = windowStore.fetch(
+ key,
+ key,
+ timestamp - 2 * windows.timeDifferenceMs(),
+ // to catch the current record's right window, if it exists, without more calls to the store
+ timestamp + 1)
+ ) {
+ KeyValue<Windowed<K>, ValueAndTimestamp<Agg>> next;
Review comment:
Aha, that makes sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a change in pull request #9039: KAFKA-5636: SlidingWindows (KIP-450)
Posted by GitBox <gi...@apache.org>.
mjsax commented on a change in pull request #9039:
URL: https://github.com/apache/kafka/pull/9039#discussion_r466028866
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/KStreamAggregationIntegrationTest.java
##########
@@ -459,6 +460,95 @@ public void shouldGroupByKey() throws Exception {
)));
}
+
+
+ @Test
+ public void shouldReduceSlidingWindows() throws Exception {
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 0);
+ final long firstBatchTimestamp = 2000L;
+ final long timeDifference = 1000L;
+ produceMessages(firstBatchTimestamp);
+ final long secondBatchTimestamp = firstBatchTimestamp + timeDifference / 2;
+ produceMessages(secondBatchTimestamp);
+ final long thirdBatchTimestamp = secondBatchTimestamp + timeDifference - 100L;
+ produceMessages(thirdBatchTimestamp);
Review comment:
I guess testing both cases would be good. Even if testing the former (fall outside of all existing windows) was my original intent.
And thank for comment Sophie: I tend to forget that we should produce all (non-empty) right windows already upfront/eagerly (and not delayed/lazily when stream-time advances beyond window-end time). In any case, it seems to be a good test case to make sure we don't (re-)emit an (unexpected) window if stream-time jumps ahead?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org