You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "jiexray (via GitHub)" <gi...@apache.org> on 2025/01/16 07:55:12 UTC
[PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
jiexray opened a new pull request, #25996:
URL: https://github.com/apache/flink/pull/25996
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Implement DataStream WindowOperator to support async state access.
## Brief change log
TODO
## Verifying this change
- UT: `AsyncWindowOperatorTest`
- IT: `testAsyncWindowWordCount` & `AsyncEventTimeWindowCheckpointingITCase`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on PR #25996:
URL: https://github.com/apache/flink/pull/25996#issuecomment-2599580729
@flinkbot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918060154
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector = new TimestampedCollector<>(output);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ keyDeclaredVariable =
+ declarationContext.declareVariable(
+ keySerializer, "_AsyncWindowOperator$key", keySerializer::createInstance);
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(keyDeclaredVariable, windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ stateWrapWithNamespace(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
Review Comment:
Maybe should add `StateFutureUtils.completedFuture(true);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918443143
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/functions/InternalAggregateProcessAsyncWindowFunction.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing.functions;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.functions.WrappingFunction;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.state.v2.StateIterator;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.util.Collector;
+
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Internal window function for wrapping a {@link ProcessWindowFunction} that takes an {@code
+ * Iterable} and an {@link AggregateFunction}. This is for async window operator.
+ *
+ * @param <K> The key type
+ * @param <W> The window type
+ * @param <T> The type of the input to the AggregateFunction
+ * @param <ACC> The type of the AggregateFunction's accumulator
+ * @param <V> The type of the AggregateFunction's result, and the input to the WindowFunction
+ * @param <R> The result type of the WindowFunction
+ */
+public final class InternalAggregateProcessAsyncWindowFunction<T, ACC, V, R, K, W extends Window>
Review Comment:
This function will be used sooner in `WindowStream` initialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918332729
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector = new TimestampedCollector<>(output);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ keyDeclaredVariable =
+ declarationContext.declareVariable(
+ keySerializer, "_AsyncWindowOperator$key", keySerializer::createInstance);
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(keyDeclaredVariable, windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ stateWrapWithNamespace(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+ keyDeclaredVariable.set(key);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenCompose(
+ ignore -> {
+ if (triggerResult.get().isFire()) {
+ return windowState
+ .asyncGet()
+ .thenCompose(
+ contents -> {
+ if (contents != null) {
+ return emitWindowContents(
+ key, window,
+ contents);
+ } else {
+ return StateFutureUtils
+ .completedFuture(
+ null);
+ }
+ });
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenCompose(
+ empty -> {
+ if (triggerResult.get().isPurge()) {
+ return windowState.asyncClear();
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onProcessingTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (!windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public OperatorAttributes getOperatorAttributes() {
+ boolean isOutputOnlyAfterEndOfStream =
+ windowAssigner instanceof GlobalWindows
+ && trigger instanceof AsyncEndOfStreamTrigger;
+ return new OperatorAttributesBuilder()
+ .setOutputOnlyAfterEndOfStream(isOutputOnlyAfterEndOfStream)
+ .build();
+ }
+
+ /**
+ * Drops all state for the given window and calls {@link AsyncTrigger#clear(Window,
+ * AsyncTrigger.TriggerContext)}.
+ *
+ * <p>The caller must ensure that the correct key is set in the state backend and the
+ * triggerContext object.
+ */
+ private StateFuture<Void> clearAllState(W window, AppendingState<IN, ACC, ACC> windowState) {
+ return windowState
+ .asyncClear()
+ .thenCompose(ignore -> triggerContext.clear())
+ .thenCompose(
+ ignore -> {
+ windowDeclaredVariable.set(window);
+ return processContext.clear();
+ });
+ }
+
+ /** Emits the contents of the given window using the {@link InternalWindowFunction}. */
+ @SuppressWarnings("unchecked")
+ private StateFuture<Void> emitWindowContents(K key, W window, ACC contents) throws Exception {
+ timestampedCollector.setAbsoluteTimestamp(window.maxTimestamp());
+ windowDeclaredVariable.set(window);
+ return userFunction.process(key, window, processContext, contents, timestampedCollector);
+ }
+
+ /**
+ * Write skipped late arriving element to SideOutput.
+ *
+ * @param element skipped late arriving element to side output
+ */
+ protected void sideOutput(StreamRecord<IN> element) {
+ output.collect(lateDataOutputTag, element);
+ }
+
+ /**
+ * Returns {@code true} if the watermark is after the end timestamp plus the allowed lateness of
+ * the given window.
+ */
+ protected boolean isWindowLate(W window) {
+ return (windowAssigner.isEventTime()
+ && (cleanupTime(window) <= internalTimerService.currentWatermark()));
+ }
+
+ /**
+ * Decide if a record is currently late, based on current watermark and allowed lateness.
+ *
+ * @param element The element to check
+ * @return The element for which should be considered when sideoutputs
+ */
+ protected boolean isElementLate(StreamRecord<IN> element) {
+ return (windowAssigner.isEventTime())
+ && (element.getTimestamp() + allowedLateness
+ <= internalTimerService.currentWatermark());
+ }
+
+ /**
+ * Registers a timer to cleanup the content of the window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void registerCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // don't set a GC timer for "end of time"
+ return;
+ }
+
+ if (windowAssigner.isEventTime()) {
+ triggerContext.registerEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.registerProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Deletes the cleanup timer set for the contents of the provided window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void deleteCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // no need to clean up because we didn't set one
+ return;
+ }
+ if (windowAssigner.isEventTime()) {
+ triggerContext.deleteEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.deleteProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Returns the cleanup time for a window, which is {@code window.maxTimestamp +
+ * allowedLateness}. In case this leads to a value greater than {@link Long#MAX_VALUE} then a
+ * cleanup time of {@link Long#MAX_VALUE} is returned.
+ *
+ * @param window the window whose cleanup time we are computing.
+ */
+ private long cleanupTime(W window) {
+ if (windowAssigner.isEventTime()) {
+ long cleanupTime = window.maxTimestamp() + allowedLateness;
+ return cleanupTime >= window.maxTimestamp() ? cleanupTime : Long.MAX_VALUE;
+ } else {
+ return window.maxTimestamp();
+ }
+ }
+
+ /** Returns {@code true} if the given time is the cleanup time for the given window. */
+ protected final boolean isCleanupTime(W window, long time) {
+ return time == cleanupTime(window);
+ }
+
+ /**
+ * Base class for per-window {@link KeyedStateStore KeyedStateStores}. Used to allow per-window
+ * state access for {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public abstract class AbstractPerWindowStateStore extends DefaultKeyedStateStore {
+
+ // we have this in the base class even though it's not used in MergingKeyStore so that
+ // we can always set it and ignore what actual implementation we have
+ protected final DeclaredVariable<W> window;
+
+ public AbstractPerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(
+ keyedStateBackend,
+ new SerializerFactory() {
+ @Override
+ public <T> TypeSerializer<T> createSerializer(
+ TypeInformation<T> typeInformation) {
+ return typeInformation.createSerializer(
+ executionConfig.getSerializerConfig());
+ }
+ });
+ this.window = window;
+ }
+ }
+
+ /**
+ * Regular per-window state store for use with {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public class PerWindowStateStore extends AbstractPerWindowStateStore {
+
+ public PerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(keyedStateBackend, executionConfig, window);
+ }
+
+ @Override
+ protected <S extends State, SV> S getPartitionedState(StateDescriptor<SV> stateDescriptor)
+ throws Exception {
+ return stateWrapWithNamespace(
+ asyncKeyedStateBackend.getOrCreateKeyedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ }
+ }
+
+ /**
+ * A utility class for handling {@code ProcessWindowFunction} invocations. This can be reused by
+ * setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code WindowContext}.
+ */
+ public class WindowContext implements InternalAsyncWindowFunction.InternalWindowContext {
+ protected final DeclaredVariable<W> window;
+
+ protected AbstractPerWindowStateStore windowState;
+
+ public WindowContext(DeclaredVariable<W> window) {
+ this.window = window;
+ // TODO: support merging window.
+ this.windowState =
+ new PerWindowStateStore(
+ getAsyncKeyedStateBackend(), getExecutionConfig(), window);
+ }
+
+ @Override
+ public String toString() {
+ return "WindowContext{Window = " + window.toString() + "}";
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return userFunction.clear(window.get(), this);
+ }
+
+ @Override
+ public long currentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public long currentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @Override
+ public KeyedStateStore windowState() {
+ return this.windowState;
+ }
+
+ @Override
+ public KeyedStateStore globalState() {
+ return AsyncWindowOperator.this.getKeyedStateStore();
+ }
+
+ @Override
+ public <X> void output(OutputTag<X> outputTag, X value) {
+ if (outputTag == null) {
+ throw new IllegalArgumentException("OutputTag must not be null.");
+ }
+ output.collect(outputTag, new StreamRecord<>(value, window.get().maxTimestamp()));
+ }
+ }
+
+ /**
+ * {@code Context} is a utility for handling {@code AsyncTrigger} invocations. It can be reused
+ * by setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code Context}
+ */
+ public class Context implements AsyncTrigger.OnMergeContext {
+ protected DeclaredVariable<K> key;
+ protected DeclaredVariable<W> window;
+
+ protected Collection<W> mergedWindows;
+
+ public Context(DeclaredVariable<K> key, DeclaredVariable<W> window) {
+ this.key = key;
+ this.window = window;
+ }
+
+ @Override
+ public MetricGroup getMetricGroup() {
+ return AsyncWindowOperator.this.getMetricGroup();
+ }
+
+ public long getCurrentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @SuppressWarnings("unchecked")
+ public <T, S extends State> S getPartitionedState(StateDescriptor<T> stateDescriptor) {
+ try {
+ return stateWrapWithNamespace(
+ AsyncWindowOperator.this.getAsyncPartitionedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ } catch (Exception e) {
+ throw new RuntimeException("Could not retrieve state", e);
+ }
+ }
+
+ @Override
+ public <T> void mergePartitionedState(StateDescriptor<T> stateDescriptor) {
+ throw new UnsupportedOperationException("Merging window not supported yet");
+ }
+
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public void registerProcessingTimeTimer(long time) {
+ internalTimerService.registerProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void registerEventTimeTimer(long time) {
+ internalTimerService.registerEventTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteProcessingTimeTimer(long time) {
+ internalTimerService.deleteProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteEventTimeTimer(long time) {
+ internalTimerService.deleteEventTimeTimer(window.get(), time);
+ }
+
+ public StateFuture<TriggerResult> onElement(StreamRecord<IN> element) throws Exception {
+ return trigger.onElement(
+ element.getValue(), element.getTimestamp(), window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onProcessingTime(long time) throws Exception {
+ return trigger.onProcessingTime(time, window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onEventTime(long time) throws Exception {
+ return trigger.onEventTime(time, window.get(), this);
+ }
+
+ public void onMerge(Collection<W> mergedWindows) throws Exception {
+ throw new UnsupportedOperationException("Merging window not support yet");
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return trigger.clear(window.get(), this);
+ }
+
+ @Override
+ public String toString() {
+ return "Context{" + "key=" + key + ", window=" + window + '}';
+ }
+ }
+
+ /** Internal class for keeping track of in-flight timers. */
+ protected static class Timer<K, W extends Window> implements Comparable<Timer<K, W>> {
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1919476870
##########
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/WindowWordCount.java:
##########
@@ -118,15 +129,33 @@ public static void main(String[] args) throws Exception {
// Using a keyBy allows performing aggregations and other
// stateful transformations over data on a per-key basis.
// This is similar to a GROUP BY clause in a SQL query.
- .keyBy(value -> value.f0)
- // create windows of windowSize records slided every slideSize records
- .countWindow(windowSize, slideSize)
- // For each key, we perform a simple sum of the "1" field, the count.
- // If the input data set is bounded, sum will output a final count for
- // each word. If it is unbounded, it will continuously output updates
- // each time it sees a new instance of each word in the stream.
- .sum(1)
- .name("counter");
+ .keyBy(value -> value.f0);
+
+ DataStream<Tuple2<String, Integer>> counts;
+ if (params.isAsyncState()) {
+ counts =
+ keyedStream
+ .enableAsyncState()
+ // create windows of windowSize records slided every slideSize records
+ .asyncCountWindow(windowSize, slideSize)
Review Comment:
Ah.... we won't provide new API for user. Only `countWindow` combined with `enableAsyncState` is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918060154
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector = new TimestampedCollector<>(output);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ keyDeclaredVariable =
+ declarationContext.declareVariable(
+ keySerializer, "_AsyncWindowOperator$key", keySerializer::createInstance);
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(keyDeclaredVariable, windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ stateWrapWithNamespace(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
Review Comment:
Maybe should add `StateFutureUtils.completedFuture(true);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920956433
##########
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/WindowWordCount.java:
##########
@@ -88,6 +93,12 @@ public static void main(String[] args) throws Exception {
// available in the Flink UI.
env.getConfig().setGlobalJobParameters(params);
+ if (params.isAsyncState()) {
+ Configuration config = Configuration.fromMap(env.getConfiguration().toMap());
+ config.set(StateBackendOptions.STATE_BACKEND, FORST_STATE_BACKEND_NAME);
+ env.configure(config);
Review Comment:
Resolved, After change:
```
// For async state, by default we will use the forst state backend.
if (params.isAsyncState()) {
Configuration config = Configuration.fromMap(env.getConfiguration().toMap());
if (!config.containsKey(StateBackendOptions.STATE_BACKEND.key())) {
config.set(StateBackendOptions.STATE_BACKEND, FORST_STATE_BACKEND_NAME);
env.configure(config);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917936589
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/functions/InternalAggregateProcessAsyncWindowFunction.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing.functions;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.functions.WrappingFunction;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.state.v2.StateIterator;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.util.Collector;
+
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Internal window function for wrapping a {@link ProcessWindowFunction} that takes an {@code
+ * Iterable} and an {@link AggregateFunction}. This is for async window operator.
+ *
+ * @param <K> The key type
+ * @param <W> The window type
+ * @param <T> The type of the input to the AggregateFunction
+ * @param <ACC> The type of the AggregateFunction's accumulator
+ * @param <V> The type of the AggregateFunction's result, and the input to the WindowFunction
+ * @param <R> The result type of the WindowFunction
+ */
+public final class InternalAggregateProcessAsyncWindowFunction<T, ACC, V, R, K, W extends Window>
+ extends WrappingFunction<ProcessWindowFunction<V, R, K, W>>
+ implements InternalAsyncWindowFunction<StateIterator<T>, R, K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final AggregateFunction<T, ACC, V> aggFunction;
+
+ public InternalAggregateProcessAsyncWindowFunction(
+ AggregateFunction<T, ACC, V> aggFunction,
+ ProcessWindowFunction<V, R, K, W> windowFunction) {
+ super(windowFunction);
+ this.aggFunction = aggFunction;
+ }
+
+ @Override
+ public StateFuture<Void> process(
+ K key,
+ final W window,
+ final InternalWindowContext context,
+ StateIterator<T> input,
+ Collector<R> out)
+ throws Exception {
+ InternalProcessWindowContext<V, R, K, W> ctx =
+ new InternalProcessWindowContext<>(wrappedFunction);
+ ctx.window = window;
+ ctx.internalContext = context;
+ AtomicReference<ACC> finalAcc = new AtomicReference<>(aggFunction.createAccumulator());
+
+ return input.onNext(
+ val -> {
+ finalAcc.set(aggFunction.add(val, finalAcc.get()));
+ })
+ .thenAccept(
+ ignore -> {
+ ProcessWindowFunction<V, R, K, W> wrappedFunction =
+ this.wrappedFunction;
+ wrappedFunction.process(
+ key,
+ ctx,
+ Collections.singletonList(
+ aggFunction.getResult(finalAcc.get())),
+ out);
+ });
+ }
+
+ @Override
+ public StateFuture<Void> clear(final W window, final InternalWindowContext context)
+ throws Exception {
+ InternalProcessWindowContext<V, R, K, W> ctx =
+ new InternalProcessWindowContext<>(wrappedFunction);
+ ctx.window = window;
+ ctx.internalContext = context;
+
+ ProcessWindowFunction<V, R, K, W> wrappedFunction = this.wrappedFunction;
+ wrappedFunction.clear(ctx);
+ return StateFutureUtils.completedFuture(null);
Review Comment:
nit. Replace all `StateFutureUtils.completedFuture(null);` to `StateFutureUtils.completedVoidFuture();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920137914
##########
flink-core/src/main/java/org/apache/flink/core/state/StateFutureUtils.java:
##########
@@ -34,6 +35,14 @@
*/
@Experimental
public class StateFutureUtils {
+ /**
+ * Returns a function that always returns a completed future that does nothing and return null.
+ */
+ public static <T>
+ FunctionWithException<T, StateFuture<T>, Exception> createVoidReturnFunction() {
Review Comment:
This util function is no use to me anymore, and I have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920957468
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers/AsyncContinuousEventTimeTrigger.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.windowing.triggers;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.common.state.v2.ReducingState;
+import org.apache.flink.api.common.state.v2.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+
+import java.time.Duration;
+
+/**
+ * A {@link Trigger} that continuously fires based on a given time interval. This fires based on
+ * {@link org.apache.flink.streaming.api.watermark.Watermark Watermarks}.
+ *
+ * @param <W> The type of {@link Window Windows} on which this trigger can operate.
+ * @see org.apache.flink.streaming.api.watermark.Watermark
+ */
+@PublicEvolving
+public class AsyncContinuousEventTimeTrigger<W extends Window> extends AsyncTrigger<Object, W> {
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920956871
##########
flink-runtime/src/main/java/org/apache/flink/streaming/api/windowing/assigners/WindowAssigner.java:
##########
@@ -64,6 +66,13 @@ public abstract Collection<W> assignWindows(
*/
public abstract Trigger<T, W> getDefaultTrigger();
+ /** Returns the default async trigger associated with this {@code WindowAssigner}. */
+ @Experimental
+ public AsyncTrigger<T, W> getDefaultAsyncTrigger() {
+ // Subclass should override this method if they support async state processing.
+ throw new RuntimeException("The assigner does not support async state.");
+ }
+
Review Comment:
Resolved.
Yes, `getDefaultAsyncTrigger()` is no use, and I have removed it from `WindowAssigner`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly merged PR #25996:
URL: https://github.com/apache/flink/pull/25996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917914164
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
+ * setCurrentNamespace} when we are going to call async state API.
+ */
+@Internal
+public class StateWithDeclaredNamespace<N> {
+ private final InternalPartitionedState<N> state;
+ private final DeclaredVariable<N> declaredNamespace;
+
+ public StateWithDeclaredNamespace(State state, DeclaredVariable<N> declaredNamespace) {
+ if (state instanceof InternalPartitionedState) {
+ this.state = (InternalPartitionedState<N>) state;
+ } else {
+ this.state = null;
+ }
+ this.declaredNamespace = declaredNamespace;
+ }
+
+ protected void wrapNamespace() {
Review Comment:
A little strange here... How about naming it `resetNamespace`
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/AggregatingStateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+
+import java.util.Collection;
+
+/** Aggregating state wrapped with declared namespace. */
+class AggregatingStateWithDeclaredNamespace<K, N, IN, ACC, OUT>
+ extends StateWithDeclaredNamespace<N>
+ implements InternalAggregatingState<K, N, IN, ACC, OUT> {
+ private final InternalAggregatingState<K, N, IN, ACC, OUT> state;
+
+ public AggregatingStateWithDeclaredNamespace(
+ InternalAggregatingState<K, N, IN, ACC, OUT> state,
+ DeclaredVariable<N> declaredNamespace) {
+ super(state, declaredNamespace);
+ this.state = state;
+ }
+
+ @Override
+ public StateFuture<OUT> asyncGet() {
+ wrapNamespace();
+ return state.asyncGet();
+ }
+
+ @Override
+ public StateFuture<Void> asyncAdd(IN value) {
+ wrapNamespace();
+ return state.asyncAdd(value);
+ }
+
+ @Override
+ public StateFuture<Void> asyncClear() {
+ wrapNamespace();
+ return state.asyncClear();
+ }
+
+ @Override
+ public StateFuture<Void> asyncMergeNamespaces(N target, Collection<N> sources) {
+ wrapNamespace();
+ return state.asyncMergeNamespaces(target, sources);
+ }
+
+ @Override
+ public StateFuture<ACC> asyncGetInternal() {
+ wrapNamespace();
+ return state.asyncGetInternal();
+ }
+
+ @Override
+ public StateFuture<Void> asyncUpdateInternal(ACC valueToStore) {
+ wrapNamespace();
+ return state.asyncUpdateInternal(valueToStore);
+ }
+
+ @Override
+ public OUT get() {
+ return state.get();
+ }
+
+ @Override
+ public void add(IN value) {
+ state.add(value);
+ }
+
+ @Override
+ public void clear() {
+ state.clear();
+ }
+
+ @Override
+ public void mergeNamespaces(N target, Collection<N> sources) {
+ state.mergeNamespaces(target, sources);
+ state.mergeNamespaces(target, sources);
Review Comment:
why twice...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920137353
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,810 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.TimestampedCollectorWithDeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.create;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ Preconditions.checkNotNull(
+ getAsyncKeyedStateBackend(), "Async state backend is not properly set.");
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector =
+ new TimestampedCollectorWithDeclaredVariable<>(output, declarationContext);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ create(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ key, window,
+ contents),
+ StateFutureUtils
+ .createVoidReturnFunction()),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isPurge(),
+ ignore -> windowState.asyncClear(),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ timer.getKey(),
+ timer.getNamespace(),
+ contents),
+ StateFutureUtils.createVoidReturnFunction()),
Review Comment:
Using `thenConditionallyCompose ` is a great idea, I have transformed to `thenConditionallyCompose` with only true branch.
`StateFutureUtils.createVoidReturnFunction` has been removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918312532
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
+ * setCurrentNamespace} when we are going to call async state API.
+ */
+@Internal
+public class StateWithDeclaredNamespace<N> {
+ private final InternalPartitionedState<N> state;
+ private final DeclaredVariable<N> declaredNamespace;
+
+ public StateWithDeclaredNamespace(State state, DeclaredVariable<N> declaredNamespace) {
+ if (state instanceof InternalPartitionedState) {
+ this.state = (InternalPartitionedState<N>) state;
+ } else {
+ this.state = null;
+ }
+ this.declaredNamespace = declaredNamespace;
+ }
+
+ protected void wrapNamespace() {
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918341973
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/functions/InternalAggregateProcessAsyncWindowFunction.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing.functions;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.functions.WrappingFunction;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.state.v2.StateIterator;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.util.Collector;
+
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Internal window function for wrapping a {@link ProcessWindowFunction} that takes an {@code
+ * Iterable} and an {@link AggregateFunction}. This is for async window operator.
+ *
+ * @param <K> The key type
+ * @param <W> The window type
+ * @param <T> The type of the input to the AggregateFunction
+ * @param <ACC> The type of the AggregateFunction's accumulator
+ * @param <V> The type of the AggregateFunction's result, and the input to the WindowFunction
+ * @param <R> The result type of the WindowFunction
+ */
+public final class InternalAggregateProcessAsyncWindowFunction<T, ACC, V, R, K, W extends Window>
+ extends WrappingFunction<ProcessWindowFunction<V, R, K, W>>
+ implements InternalAsyncWindowFunction<StateIterator<T>, R, K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private final AggregateFunction<T, ACC, V> aggFunction;
+
+ public InternalAggregateProcessAsyncWindowFunction(
+ AggregateFunction<T, ACC, V> aggFunction,
+ ProcessWindowFunction<V, R, K, W> windowFunction) {
+ super(windowFunction);
+ this.aggFunction = aggFunction;
+ }
+
+ @Override
+ public StateFuture<Void> process(
+ K key,
+ final W window,
+ final InternalWindowContext context,
+ StateIterator<T> input,
+ Collector<R> out)
+ throws Exception {
+ InternalProcessWindowContext<V, R, K, W> ctx =
+ new InternalProcessWindowContext<>(wrappedFunction);
+ ctx.window = window;
+ ctx.internalContext = context;
+ AtomicReference<ACC> finalAcc = new AtomicReference<>(aggFunction.createAccumulator());
+
+ return input.onNext(
+ val -> {
+ finalAcc.set(aggFunction.add(val, finalAcc.get()));
+ })
+ .thenAccept(
+ ignore -> {
+ ProcessWindowFunction<V, R, K, W> wrappedFunction =
+ this.wrappedFunction;
+ wrappedFunction.process(
+ key,
+ ctx,
+ Collections.singletonList(
+ aggFunction.getResult(finalAcc.get())),
+ out);
+ });
+ }
+
+ @Override
+ public StateFuture<Void> clear(final W window, final InternalWindowContext context)
+ throws Exception {
+ InternalProcessWindowContext<V, R, K, W> ctx =
+ new InternalProcessWindowContext<>(wrappedFunction);
+ ctx.window = window;
+ ctx.internalContext = context;
+
+ ProcessWindowFunction<V, R, K, W> wrappedFunction = this.wrappedFunction;
+ wrappedFunction.clear(ctx);
+ return StateFutureUtils.completedFuture(null);
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918436120
##########
flink-runtime/src/main/java/org/apache/flink/streaming/api/windowing/assigners/WindowAssigner.java:
##########
@@ -64,6 +65,12 @@ public abstract Collection<W> assignWindows(
*/
public abstract Trigger<T, W> getDefaultTrigger();
+ /** Returns the default async trigger associated with this {@code WindowAssigner}. */
+ public AsyncTrigger<T, W> getDefaultAsyncTrigger() {
Review Comment:
WindowAssigner is stateless, I think it is not a must to introudce a async version.
Differently, Trigger is bind with state operation, and I need to introduce an async Trigger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917925184
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
Review Comment:
Suggestion:
A partitioned state that wraps a declared namespace and hide the namespace switching from user. User will use the state just like the public state APIs without any namespaces manipulation. Useful in window operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917988381
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
Review Comment:
Seems unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918104714
##########
flink-runtime/src/main/java/org/apache/flink/streaming/api/windowing/assigners/WindowAssigner.java:
##########
@@ -64,6 +65,12 @@ public abstract Collection<W> assignWindows(
*/
public abstract Trigger<T, W> getDefaultTrigger();
+ /** Returns the default async trigger associated with this {@code WindowAssigner}. */
+ public AsyncTrigger<T, W> getDefaultAsyncTrigger() {
Review Comment:
And is it better to introduce a async version of `WindowAssigner ` or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #25996:
URL: https://github.com/apache/flink/pull/25996#issuecomment-2595002635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "d442af5859312a03adf2491bdbfbb98f52e9ce8a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d442af5859312a03adf2491bdbfbb98f52e9ce8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d442af5859312a03adf2491bdbfbb98f52e9ce8a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918306337
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/ValueStateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/** Value state wrapped with declared namespace. */
+@Internal
+class ValueStateWithDeclaredNamespace<K, N, V> extends StateWithDeclaredNamespace<N>
+ implements InternalValueState<K, N, V> {
+ private final InternalValueState<K, N, V> state;
+
+ public ValueStateWithDeclaredNamespace(
+ InternalValueState<K, N, V> state, DeclaredVariable<N> declaredNamespace) {
+ super(state, declaredNamespace);
+ this.state = state;
+ }
+
+ @Override
+ public StateFuture<Void> asyncClear() {
+ wrapNamespace();
+ return state.asyncClear();
+ }
+
+ @Override
+ public StateFuture<V> asyncValue() {
+ wrapNamespace();
+ return state.asyncValue();
+ }
+
+ @Override
+ public StateFuture<Void> asyncUpdate(V value) {
+ wrapNamespace();
+ return state.asyncUpdate(value);
+ }
+
+ @Override
+ public void clear() {
+ state.clear();
+ }
+
+ @Override
+ public V value() {
+ return state.value();
+ }
+
+ @Override
+ public void update(V value) {
+ state.update(value);
+ }
+
+ @Override
+ public void setCurrentNamespace(N namespace) {
+ state.setCurrentNamespace(namespace);
Review Comment:
resolved.
After let `StateWithDeclaredNamespace` implements `InternalKeyedState`, Base class `StateWithDeclaredNamespace` unifies the impl of `setCurrentNamespace ()` in all XxxWithDeclaredNamespace class in:
```
@Override
public void setCurrentNamespace(N namespace) {
declaredNamespace.set(namespace);
state.setCurrentNamespace(namespace);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918334489
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918338304
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
Review Comment:
Resolved. I have removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917941066
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/functions/InternalAggregateProcessAsyncWindowFunction.java:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing.functions;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.IterationRuntimeContext;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.functions.WrappingFunction;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.state.v2.StateIterator;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.util.Collector;
+
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Internal window function for wrapping a {@link ProcessWindowFunction} that takes an {@code
+ * Iterable} and an {@link AggregateFunction}. This is for async window operator.
+ *
+ * @param <K> The key type
+ * @param <W> The window type
+ * @param <T> The type of the input to the AggregateFunction
+ * @param <ACC> The type of the AggregateFunction's accumulator
+ * @param <V> The type of the AggregateFunction's result, and the input to the WindowFunction
+ * @param <R> The result type of the WindowFunction
+ */
+public final class InternalAggregateProcessAsyncWindowFunction<T, ACC, V, R, K, W extends Window>
Review Comment:
It seems this function is not used for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on PR #25996:
URL: https://github.com/apache/flink/pull/25996#issuecomment-2595672230
@Zakelly Thanks for reviewing. I have fixed the comments.
Moreover, I have added another two commits to run an end-to-end example (i.e., WindowWordCount) for window datastream with async state API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920951003
##########
flink-runtime/src/main/java/org/apache/flink/streaming/api/windowing/assigners/WindowAssigner.java:
##########
@@ -64,6 +66,13 @@ public abstract Collection<W> assignWindows(
*/
public abstract Trigger<T, W> getDefaultTrigger();
+ /** Returns the default async trigger associated with this {@code WindowAssigner}. */
+ @Experimental
+ public AsyncTrigger<T, W> getDefaultAsyncTrigger() {
+ // Subclass should override this method if they support async state processing.
+ throw new RuntimeException("The assigner does not support async state.");
+ }
+
Review Comment:
Seems useless for now? Can we remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918001457
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector = new TimestampedCollector<>(output);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ keyDeclaredVariable =
+ declarationContext.declareVariable(
+ keySerializer, "_AsyncWindowOperator$key", keySerializer::createInstance);
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(keyDeclaredVariable, windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ stateWrapWithNamespace(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+ keyDeclaredVariable.set(key);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenCompose(
+ ignore -> {
+ if (triggerResult.get().isFire()) {
+ return windowState
+ .asyncGet()
+ .thenCompose(
+ contents -> {
+ if (contents != null) {
+ return emitWindowContents(
+ key, window,
+ contents);
+ } else {
+ return StateFutureUtils
+ .completedFuture(
+ null);
+ }
+ });
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenCompose(
+ empty -> {
+ if (triggerResult.get().isPurge()) {
+ return windowState.asyncClear();
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onProcessingTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (!windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public OperatorAttributes getOperatorAttributes() {
+ boolean isOutputOnlyAfterEndOfStream =
+ windowAssigner instanceof GlobalWindows
+ && trigger instanceof AsyncEndOfStreamTrigger;
+ return new OperatorAttributesBuilder()
+ .setOutputOnlyAfterEndOfStream(isOutputOnlyAfterEndOfStream)
+ .build();
+ }
+
+ /**
+ * Drops all state for the given window and calls {@link AsyncTrigger#clear(Window,
+ * AsyncTrigger.TriggerContext)}.
+ *
+ * <p>The caller must ensure that the correct key is set in the state backend and the
+ * triggerContext object.
+ */
+ private StateFuture<Void> clearAllState(W window, AppendingState<IN, ACC, ACC> windowState) {
+ return windowState
+ .asyncClear()
+ .thenCompose(ignore -> triggerContext.clear())
+ .thenCompose(
+ ignore -> {
+ windowDeclaredVariable.set(window);
+ return processContext.clear();
+ });
+ }
+
+ /** Emits the contents of the given window using the {@link InternalWindowFunction}. */
+ @SuppressWarnings("unchecked")
+ private StateFuture<Void> emitWindowContents(K key, W window, ACC contents) throws Exception {
+ timestampedCollector.setAbsoluteTimestamp(window.maxTimestamp());
+ windowDeclaredVariable.set(window);
+ return userFunction.process(key, window, processContext, contents, timestampedCollector);
+ }
+
+ /**
+ * Write skipped late arriving element to SideOutput.
+ *
+ * @param element skipped late arriving element to side output
+ */
+ protected void sideOutput(StreamRecord<IN> element) {
+ output.collect(lateDataOutputTag, element);
+ }
+
+ /**
+ * Returns {@code true} if the watermark is after the end timestamp plus the allowed lateness of
+ * the given window.
+ */
+ protected boolean isWindowLate(W window) {
+ return (windowAssigner.isEventTime()
+ && (cleanupTime(window) <= internalTimerService.currentWatermark()));
+ }
+
+ /**
+ * Decide if a record is currently late, based on current watermark and allowed lateness.
+ *
+ * @param element The element to check
+ * @return The element for which should be considered when sideoutputs
+ */
+ protected boolean isElementLate(StreamRecord<IN> element) {
+ return (windowAssigner.isEventTime())
+ && (element.getTimestamp() + allowedLateness
+ <= internalTimerService.currentWatermark());
+ }
+
+ /**
+ * Registers a timer to cleanup the content of the window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void registerCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // don't set a GC timer for "end of time"
+ return;
+ }
+
+ if (windowAssigner.isEventTime()) {
+ triggerContext.registerEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.registerProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Deletes the cleanup timer set for the contents of the provided window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void deleteCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // no need to clean up because we didn't set one
+ return;
+ }
+ if (windowAssigner.isEventTime()) {
+ triggerContext.deleteEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.deleteProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Returns the cleanup time for a window, which is {@code window.maxTimestamp +
+ * allowedLateness}. In case this leads to a value greater than {@link Long#MAX_VALUE} then a
+ * cleanup time of {@link Long#MAX_VALUE} is returned.
+ *
+ * @param window the window whose cleanup time we are computing.
+ */
+ private long cleanupTime(W window) {
+ if (windowAssigner.isEventTime()) {
+ long cleanupTime = window.maxTimestamp() + allowedLateness;
+ return cleanupTime >= window.maxTimestamp() ? cleanupTime : Long.MAX_VALUE;
+ } else {
+ return window.maxTimestamp();
+ }
+ }
+
+ /** Returns {@code true} if the given time is the cleanup time for the given window. */
+ protected final boolean isCleanupTime(W window, long time) {
+ return time == cleanupTime(window);
+ }
+
+ /**
+ * Base class for per-window {@link KeyedStateStore KeyedStateStores}. Used to allow per-window
+ * state access for {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public abstract class AbstractPerWindowStateStore extends DefaultKeyedStateStore {
+
+ // we have this in the base class even though it's not used in MergingKeyStore so that
+ // we can always set it and ignore what actual implementation we have
+ protected final DeclaredVariable<W> window;
+
+ public AbstractPerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(
+ keyedStateBackend,
+ new SerializerFactory() {
+ @Override
+ public <T> TypeSerializer<T> createSerializer(
+ TypeInformation<T> typeInformation) {
+ return typeInformation.createSerializer(
+ executionConfig.getSerializerConfig());
+ }
+ });
+ this.window = window;
+ }
+ }
+
+ /**
+ * Regular per-window state store for use with {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public class PerWindowStateStore extends AbstractPerWindowStateStore {
+
+ public PerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(keyedStateBackend, executionConfig, window);
+ }
+
+ @Override
+ protected <S extends State, SV> S getPartitionedState(StateDescriptor<SV> stateDescriptor)
+ throws Exception {
+ return stateWrapWithNamespace(
+ asyncKeyedStateBackend.getOrCreateKeyedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ }
+ }
+
+ /**
+ * A utility class for handling {@code ProcessWindowFunction} invocations. This can be reused by
+ * setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code WindowContext}.
+ */
+ public class WindowContext implements InternalAsyncWindowFunction.InternalWindowContext {
+ protected final DeclaredVariable<W> window;
+
+ protected AbstractPerWindowStateStore windowState;
+
+ public WindowContext(DeclaredVariable<W> window) {
+ this.window = window;
+ // TODO: support merging window.
+ this.windowState =
+ new PerWindowStateStore(
+ getAsyncKeyedStateBackend(), getExecutionConfig(), window);
+ }
+
+ @Override
+ public String toString() {
+ return "WindowContext{Window = " + window.toString() + "}";
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return userFunction.clear(window.get(), this);
+ }
+
+ @Override
+ public long currentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public long currentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @Override
+ public KeyedStateStore windowState() {
+ return this.windowState;
+ }
+
+ @Override
+ public KeyedStateStore globalState() {
+ return AsyncWindowOperator.this.getKeyedStateStore();
+ }
+
+ @Override
+ public <X> void output(OutputTag<X> outputTag, X value) {
+ if (outputTag == null) {
+ throw new IllegalArgumentException("OutputTag must not be null.");
+ }
+ output.collect(outputTag, new StreamRecord<>(value, window.get().maxTimestamp()));
+ }
+ }
+
+ /**
+ * {@code Context} is a utility for handling {@code AsyncTrigger} invocations. It can be reused
+ * by setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code Context}
+ */
+ public class Context implements AsyncTrigger.OnMergeContext {
+ protected DeclaredVariable<K> key;
+ protected DeclaredVariable<W> window;
+
+ protected Collection<W> mergedWindows;
+
+ public Context(DeclaredVariable<K> key, DeclaredVariable<W> window) {
+ this.key = key;
+ this.window = window;
+ }
+
+ @Override
+ public MetricGroup getMetricGroup() {
+ return AsyncWindowOperator.this.getMetricGroup();
+ }
+
+ public long getCurrentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @SuppressWarnings("unchecked")
+ public <T, S extends State> S getPartitionedState(StateDescriptor<T> stateDescriptor) {
+ try {
+ return stateWrapWithNamespace(
+ AsyncWindowOperator.this.getAsyncPartitionedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ } catch (Exception e) {
+ throw new RuntimeException("Could not retrieve state", e);
+ }
+ }
+
+ @Override
+ public <T> void mergePartitionedState(StateDescriptor<T> stateDescriptor) {
+ throw new UnsupportedOperationException("Merging window not supported yet");
+ }
+
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public void registerProcessingTimeTimer(long time) {
+ internalTimerService.registerProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void registerEventTimeTimer(long time) {
+ internalTimerService.registerEventTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteProcessingTimeTimer(long time) {
+ internalTimerService.deleteProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteEventTimeTimer(long time) {
+ internalTimerService.deleteEventTimeTimer(window.get(), time);
+ }
+
+ public StateFuture<TriggerResult> onElement(StreamRecord<IN> element) throws Exception {
+ return trigger.onElement(
+ element.getValue(), element.getTimestamp(), window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onProcessingTime(long time) throws Exception {
+ return trigger.onProcessingTime(time, window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onEventTime(long time) throws Exception {
+ return trigger.onEventTime(time, window.get(), this);
+ }
+
+ public void onMerge(Collection<W> mergedWindows) throws Exception {
+ throw new UnsupportedOperationException("Merging window not support yet");
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return trigger.clear(window.get(), this);
+ }
+
+ @Override
+ public String toString() {
+ return "Context{" + "key=" + key + ", window=" + window + '}';
+ }
+ }
+
+ /** Internal class for keeping track of in-flight timers. */
+ protected static class Timer<K, W extends Window> implements Comparable<Timer<K, W>> {
Review Comment:
Could this class be removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918029036
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
Review Comment:
Please use `TimestampedCollectorWithDeclaredVariable` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918291061
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/AggregatingStateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+
+import java.util.Collection;
+
+/** Aggregating state wrapped with declared namespace. */
+class AggregatingStateWithDeclaredNamespace<K, N, IN, ACC, OUT>
+ extends StateWithDeclaredNamespace<N>
+ implements InternalAggregatingState<K, N, IN, ACC, OUT> {
+ private final InternalAggregatingState<K, N, IN, ACC, OUT> state;
+
+ public AggregatingStateWithDeclaredNamespace(
+ InternalAggregatingState<K, N, IN, ACC, OUT> state,
+ DeclaredVariable<N> declaredNamespace) {
+ super(state, declaredNamespace);
+ this.state = state;
+ }
+
+ @Override
+ public StateFuture<OUT> asyncGet() {
+ wrapNamespace();
+ return state.asyncGet();
+ }
+
+ @Override
+ public StateFuture<Void> asyncAdd(IN value) {
+ wrapNamespace();
+ return state.asyncAdd(value);
+ }
+
+ @Override
+ public StateFuture<Void> asyncClear() {
+ wrapNamespace();
+ return state.asyncClear();
+ }
+
+ @Override
+ public StateFuture<Void> asyncMergeNamespaces(N target, Collection<N> sources) {
+ wrapNamespace();
+ return state.asyncMergeNamespaces(target, sources);
+ }
+
+ @Override
+ public StateFuture<ACC> asyncGetInternal() {
+ wrapNamespace();
+ return state.asyncGetInternal();
+ }
+
+ @Override
+ public StateFuture<Void> asyncUpdateInternal(ACC valueToStore) {
+ wrapNamespace();
+ return state.asyncUpdateInternal(valueToStore);
+ }
+
+ @Override
+ public OUT get() {
+ return state.get();
+ }
+
+ @Override
+ public void add(IN value) {
+ state.add(value);
+ }
+
+ @Override
+ public void clear() {
+ state.clear();
+ }
+
+ @Override
+ public void mergeNamespaces(N target, Collection<N> sources) {
+ state.mergeNamespaces(target, sources);
+ state.mergeNamespaces(target, sources);
Review Comment:
Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1919497038
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,810 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.TimestampedCollectorWithDeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.create;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ Preconditions.checkNotNull(
+ getAsyncKeyedStateBackend(), "Async state backend is not properly set.");
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector =
+ new TimestampedCollectorWithDeclaredVariable<>(output, declarationContext);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ create(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ key, window,
+ contents),
+ StateFutureUtils
+ .createVoidReturnFunction()),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isPurge(),
+ ignore -> windowState.asyncClear(),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ timer.getKey(),
+ timer.getNamespace(),
+ contents),
+ StateFutureUtils.createVoidReturnFunction()),
Review Comment:
Or you can use the `thenConditionallyCompose` with only true branch provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1919492658
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,810 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.TimestampedCollectorWithDeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.create;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ Preconditions.checkNotNull(
+ getAsyncKeyedStateBackend(), "Async state backend is not properly set.");
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector =
+ new TimestampedCollectorWithDeclaredVariable<>(output, declarationContext);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ create(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ key, window,
+ contents),
+ StateFutureUtils
+ .createVoidReturnFunction()),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isPurge(),
+ ignore -> windowState.asyncClear(),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ timer.getKey(),
+ timer.getNamespace(),
+ contents),
+ StateFutureUtils.createVoidReturnFunction()),
Review Comment:
Consider `import static StateFutureUtils.createVoidReturnFunction` since you are using this so many times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918101684
##########
flink-runtime/src/main/java/org/apache/flink/streaming/api/windowing/assigners/WindowAssigner.java:
##########
@@ -64,6 +65,12 @@ public abstract Collection<W> assignWindows(
*/
public abstract Trigger<T, W> getDefaultTrigger();
+ /** Returns the default async trigger associated with this {@code WindowAssigner}. */
+ public AsyncTrigger<T, W> getDefaultAsyncTrigger() {
Review Comment:
Add `@Experimental` or `@internal`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920955938
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/triggers/AsyncTrigger.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+
+import java.io.Serializable;
+
+/**
+ * A {@code Trigger} determines when a pane of a window should be evaluated to emit the results for
+ * that part of the window. This trigger is for the async window operator, i.e., {@link
+ * org.apache.flink.runtime.asyncprocessing.operators.windowing.AsyncWindowOperator}.
+ *
+ * <p>All State APIs in the async trigger are from State V2.
+ *
+ * <p>A pane is the bucket of elements that have the same key (assigned by the {@link
+ * org.apache.flink.api.java.functions.KeySelector}) and same {@link Window}. An element can be in
+ * multiple panes if it was assigned to multiple windows by the {@link
+ * org.apache.flink.streaming.api.windowing.assigners.WindowAssigner}. These panes all have their
+ * own instance of the {@code Trigger}.
+ *
+ * <p>Triggers must not maintain state internally since they can be re-created or reused for
+ * different keys. All necessary state should be persisted using the state abstraction available on
+ * the {@link TriggerContext}.
+ *
+ * <p>When used with a {@link
+ * org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner} the {@code Trigger}
+ * must return {@code true} from {@link #canMerge()} and {@link #onMerge(Window, OnMergeContext)}
+ * most be properly implemented.
+ *
+ * @param <T> The type of elements on which this {@code Trigger} works.
+ * @param <W> The type of {@link Window Windows} on which this {@code Trigger} can operate.
+ */
+@Experimental
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920950253
##########
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/WindowWordCount.java:
##########
@@ -88,6 +93,12 @@ public static void main(String[] args) throws Exception {
// available in the Flink UI.
env.getConfig().setGlobalJobParameters(params);
+ if (params.isAsyncState()) {
+ Configuration config = Configuration.fromMap(env.getConfiguration().toMap());
+ config.set(StateBackendOptions.STATE_BACKEND, FORST_STATE_BACKEND_NAME);
+ env.configure(config);
Review Comment:
```suggestion
if (!config.containsKey(StateBackendOptions.STATE_BACKEND.key())) {
config.set(StateBackendOptions.STATE_BACKEND, FORST_STATE_BACKEND_NAME);
env.configure(config);
}
```
Same for the one in WordCount.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917926823
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
+ * setCurrentNamespace} when we are going to call async state API.
+ */
+@Internal
+public class StateWithDeclaredNamespace<N> {
Review Comment:
How about implement `InternalKeyedState` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917973049
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
+ * setCurrentNamespace} when we are going to call async state API.
+ */
+@Internal
+public class StateWithDeclaredNamespace<N> {
+ private final InternalPartitionedState<N> state;
+ private final DeclaredVariable<N> declaredNamespace;
+
+ public StateWithDeclaredNamespace(State state, DeclaredVariable<N> declaredNamespace) {
+ if (state instanceof InternalPartitionedState) {
+ this.state = (InternalPartitionedState<N>) state;
+ } else {
+ this.state = null;
+ }
+ this.declaredNamespace = declaredNamespace;
+ }
+
+ protected void wrapNamespace() {
+ if (this.state != null) {
+ state.setCurrentNamespace(declaredNamespace.get());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ public static <N, S extends State> S stateWrapWithNamespace(
Review Comment:
```suggestion
public static <N, S extends State> S create(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1919492658
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,810 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.TimestampedCollectorWithDeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.create;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ Preconditions.checkNotNull(
+ getAsyncKeyedStateBackend(), "Async state backend is not properly set.");
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector =
+ new TimestampedCollectorWithDeclaredVariable<>(output, declarationContext);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ create(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ key, window,
+ contents),
+ StateFutureUtils
+ .createVoidReturnFunction()),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isPurge(),
+ ignore -> windowState.asyncClear(),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ timer.getKey(),
+ timer.getNamespace(),
+ contents),
+ StateFutureUtils.createVoidReturnFunction()),
Review Comment:
Consider import static `StateFutureUtils.createVoidReturnFunction` since you are using this so many times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1919497856
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,810 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.TimestampedCollectorWithDeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.create;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ Preconditions.checkNotNull(
+ getAsyncKeyedStateBackend(), "Async state backend is not properly set.");
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector =
+ new TimestampedCollectorWithDeclaredVariable<>(output, declarationContext);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ create(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ key, window,
+ contents),
+ StateFutureUtils
+ .createVoidReturnFunction()),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isPurge(),
+ ignore -> windowState.asyncClear(),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
Review Comment:
```suggestion
.thenAccept(
empty -> {
registerCleanupTimer(window);
}));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918017015
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector = new TimestampedCollector<>(output);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ keyDeclaredVariable =
+ declarationContext.declareVariable(
+ keySerializer, "_AsyncWindowOperator$key", keySerializer::createInstance);
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(keyDeclaredVariable, windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ stateWrapWithNamespace(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+ keyDeclaredVariable.set(key);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenCompose(
+ ignore -> {
+ if (triggerResult.get().isFire()) {
+ return windowState
+ .asyncGet()
+ .thenCompose(
+ contents -> {
+ if (contents != null) {
+ return emitWindowContents(
+ key, window,
+ contents);
+ } else {
+ return StateFutureUtils
+ .completedFuture(
+ null);
+ }
+ });
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenCompose(
+ empty -> {
+ if (triggerResult.get().isPurge()) {
+ return windowState.asyncClear();
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onProcessingTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (!windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public OperatorAttributes getOperatorAttributes() {
+ boolean isOutputOnlyAfterEndOfStream =
+ windowAssigner instanceof GlobalWindows
+ && trigger instanceof AsyncEndOfStreamTrigger;
+ return new OperatorAttributesBuilder()
+ .setOutputOnlyAfterEndOfStream(isOutputOnlyAfterEndOfStream)
+ .build();
+ }
+
+ /**
+ * Drops all state for the given window and calls {@link AsyncTrigger#clear(Window,
+ * AsyncTrigger.TriggerContext)}.
+ *
+ * <p>The caller must ensure that the correct key is set in the state backend and the
+ * triggerContext object.
+ */
+ private StateFuture<Void> clearAllState(W window, AppendingState<IN, ACC, ACC> windowState) {
+ return windowState
+ .asyncClear()
+ .thenCompose(ignore -> triggerContext.clear())
+ .thenCompose(
+ ignore -> {
+ windowDeclaredVariable.set(window);
+ return processContext.clear();
+ });
+ }
+
+ /** Emits the contents of the given window using the {@link InternalWindowFunction}. */
+ @SuppressWarnings("unchecked")
+ private StateFuture<Void> emitWindowContents(K key, W window, ACC contents) throws Exception {
+ timestampedCollector.setAbsoluteTimestamp(window.maxTimestamp());
+ windowDeclaredVariable.set(window);
+ return userFunction.process(key, window, processContext, contents, timestampedCollector);
+ }
+
+ /**
+ * Write skipped late arriving element to SideOutput.
+ *
+ * @param element skipped late arriving element to side output
+ */
+ protected void sideOutput(StreamRecord<IN> element) {
+ output.collect(lateDataOutputTag, element);
+ }
+
+ /**
+ * Returns {@code true} if the watermark is after the end timestamp plus the allowed lateness of
+ * the given window.
+ */
+ protected boolean isWindowLate(W window) {
+ return (windowAssigner.isEventTime()
+ && (cleanupTime(window) <= internalTimerService.currentWatermark()));
+ }
+
+ /**
+ * Decide if a record is currently late, based on current watermark and allowed lateness.
+ *
+ * @param element The element to check
+ * @return The element for which should be considered when sideoutputs
+ */
+ protected boolean isElementLate(StreamRecord<IN> element) {
+ return (windowAssigner.isEventTime())
+ && (element.getTimestamp() + allowedLateness
+ <= internalTimerService.currentWatermark());
+ }
+
+ /**
+ * Registers a timer to cleanup the content of the window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void registerCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // don't set a GC timer for "end of time"
+ return;
+ }
+
+ if (windowAssigner.isEventTime()) {
+ triggerContext.registerEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.registerProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Deletes the cleanup timer set for the contents of the provided window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void deleteCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // no need to clean up because we didn't set one
+ return;
+ }
+ if (windowAssigner.isEventTime()) {
+ triggerContext.deleteEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.deleteProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Returns the cleanup time for a window, which is {@code window.maxTimestamp +
+ * allowedLateness}. In case this leads to a value greater than {@link Long#MAX_VALUE} then a
+ * cleanup time of {@link Long#MAX_VALUE} is returned.
+ *
+ * @param window the window whose cleanup time we are computing.
+ */
+ private long cleanupTime(W window) {
+ if (windowAssigner.isEventTime()) {
+ long cleanupTime = window.maxTimestamp() + allowedLateness;
+ return cleanupTime >= window.maxTimestamp() ? cleanupTime : Long.MAX_VALUE;
+ } else {
+ return window.maxTimestamp();
+ }
+ }
+
+ /** Returns {@code true} if the given time is the cleanup time for the given window. */
+ protected final boolean isCleanupTime(W window, long time) {
+ return time == cleanupTime(window);
+ }
+
+ /**
+ * Base class for per-window {@link KeyedStateStore KeyedStateStores}. Used to allow per-window
+ * state access for {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public abstract class AbstractPerWindowStateStore extends DefaultKeyedStateStore {
+
+ // we have this in the base class even though it's not used in MergingKeyStore so that
+ // we can always set it and ignore what actual implementation we have
+ protected final DeclaredVariable<W> window;
+
+ public AbstractPerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(
+ keyedStateBackend,
+ new SerializerFactory() {
+ @Override
+ public <T> TypeSerializer<T> createSerializer(
+ TypeInformation<T> typeInformation) {
+ return typeInformation.createSerializer(
+ executionConfig.getSerializerConfig());
+ }
+ });
+ this.window = window;
+ }
+ }
+
+ /**
+ * Regular per-window state store for use with {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public class PerWindowStateStore extends AbstractPerWindowStateStore {
+
+ public PerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(keyedStateBackend, executionConfig, window);
+ }
+
+ @Override
+ protected <S extends State, SV> S getPartitionedState(StateDescriptor<SV> stateDescriptor)
+ throws Exception {
+ return stateWrapWithNamespace(
+ asyncKeyedStateBackend.getOrCreateKeyedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ }
+ }
+
+ /**
+ * A utility class for handling {@code ProcessWindowFunction} invocations. This can be reused by
+ * setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code WindowContext}.
+ */
+ public class WindowContext implements InternalAsyncWindowFunction.InternalWindowContext {
+ protected final DeclaredVariable<W> window;
+
+ protected AbstractPerWindowStateStore windowState;
+
+ public WindowContext(DeclaredVariable<W> window) {
+ this.window = window;
+ // TODO: support merging window.
+ this.windowState =
+ new PerWindowStateStore(
+ getAsyncKeyedStateBackend(), getExecutionConfig(), window);
+ }
+
+ @Override
+ public String toString() {
+ return "WindowContext{Window = " + window.toString() + "}";
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return userFunction.clear(window.get(), this);
+ }
+
+ @Override
+ public long currentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public long currentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @Override
+ public KeyedStateStore windowState() {
+ return this.windowState;
+ }
+
+ @Override
+ public KeyedStateStore globalState() {
+ return AsyncWindowOperator.this.getKeyedStateStore();
+ }
+
+ @Override
+ public <X> void output(OutputTag<X> outputTag, X value) {
+ if (outputTag == null) {
+ throw new IllegalArgumentException("OutputTag must not be null.");
+ }
+ output.collect(outputTag, new StreamRecord<>(value, window.get().maxTimestamp()));
+ }
+ }
+
+ /**
+ * {@code Context} is a utility for handling {@code AsyncTrigger} invocations. It can be reused
+ * by setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code Context}
+ */
+ public class Context implements AsyncTrigger.OnMergeContext {
+ protected DeclaredVariable<K> key;
+ protected DeclaredVariable<W> window;
+
+ protected Collection<W> mergedWindows;
+
+ public Context(DeclaredVariable<K> key, DeclaredVariable<W> window) {
+ this.key = key;
+ this.window = window;
+ }
+
+ @Override
+ public MetricGroup getMetricGroup() {
+ return AsyncWindowOperator.this.getMetricGroup();
+ }
+
+ public long getCurrentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @SuppressWarnings("unchecked")
+ public <T, S extends State> S getPartitionedState(StateDescriptor<T> stateDescriptor) {
+ try {
+ return stateWrapWithNamespace(
+ AsyncWindowOperator.this.getAsyncPartitionedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ } catch (Exception e) {
+ throw new RuntimeException("Could not retrieve state", e);
+ }
+ }
+
+ @Override
+ public <T> void mergePartitionedState(StateDescriptor<T> stateDescriptor) {
+ throw new UnsupportedOperationException("Merging window not supported yet");
+ }
+
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public void registerProcessingTimeTimer(long time) {
+ internalTimerService.registerProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void registerEventTimeTimer(long time) {
+ internalTimerService.registerEventTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteProcessingTimeTimer(long time) {
+ internalTimerService.deleteProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteEventTimeTimer(long time) {
+ internalTimerService.deleteEventTimeTimer(window.get(), time);
+ }
+
+ public StateFuture<TriggerResult> onElement(StreamRecord<IN> element) throws Exception {
+ return trigger.onElement(
+ element.getValue(), element.getTimestamp(), window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onProcessingTime(long time) throws Exception {
+ return trigger.onProcessingTime(time, window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onEventTime(long time) throws Exception {
+ return trigger.onEventTime(time, window.get(), this);
+ }
+
+ public void onMerge(Collection<W> mergedWindows) throws Exception {
+ throw new UnsupportedOperationException("Merging window not support yet");
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return trigger.clear(window.get(), this);
+ }
+
+ @Override
+ public String toString() {
+ return "Context{" + "key=" + key + ", window=" + window + '}';
+ }
+ }
+
+ /** Internal class for keeping track of in-flight timers. */
+ protected static class Timer<K, W extends Window> implements Comparable<Timer<K, W>> {
+ protected long timestamp;
+ protected K key;
+ protected W window;
+
+ public Timer(long timestamp, K key, W window) {
+ this.timestamp = timestamp;
+ this.key = key;
+ this.window = window;
+ }
+
+ @Override
+ public int compareTo(Timer<K, W> o) {
+ return Long.compare(this.timestamp, o.timestamp);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ Timer<?, ?> timer = (Timer<?, ?>) o;
+
+ return timestamp == timer.timestamp
+ && key.equals(timer.key)
+ && window.equals(timer.window);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = (int) (timestamp ^ (timestamp >>> 32));
+ result = 31 * result + key.hashCode();
+ result = 31 * result + window.hashCode();
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "Timer{"
+ + "timestamp="
+ + timestamp
+ + ", key="
+ + key
+ + ", window="
+ + window
+ + '}';
+ }
+ }
+
+ protected <S extends State, N, T> S getAsyncPartitionedState(
+ N namespace, TypeSerializer<N> namespaceSerializer, StateDescriptor<T> stateDescriptor)
+ throws Exception {
+ return getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(namespace, namespaceSerializer, stateDescriptor);
+ }
Review Comment:
I'd suggest this method includes `stateWrapWithNamespace`. WDYT?
And no need for `Async`, just `getPartitionedState ` seems good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918440423
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918318689
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
Review Comment:
Resolved.
Changed to:
```
/**
* A partitioned state that wraps a declared namespace and hide the namespace switching from user.
* User will only use the state just like the public state APIs without any consideration on namespace.
*
* <p>This wrap class is useful in DataStream window operation, where namespace is managed by the operator
* and user function is free from namespace manipulation.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918319804
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
+ * setCurrentNamespace} when we are going to call async state API.
+ */
+@Internal
+public class StateWithDeclaredNamespace<N> {
Review Comment:
Good idea. I have changed it and let `StateWithDeclaredNamespace` unify the impl of `setCurrentNamespace(N)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918355186
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector = new TimestampedCollector<>(output);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ keyDeclaredVariable =
+ declarationContext.declareVariable(
+ keySerializer, "_AsyncWindowOperator$key", keySerializer::createInstance);
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(keyDeclaredVariable, windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ stateWrapWithNamespace(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+ keyDeclaredVariable.set(key);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenCompose(
+ ignore -> {
+ if (triggerResult.get().isFire()) {
+ return windowState
+ .asyncGet()
+ .thenCompose(
+ contents -> {
+ if (contents != null) {
+ return emitWindowContents(
+ key, window,
+ contents);
+ } else {
+ return StateFutureUtils
+ .completedFuture(
+ null);
+ }
+ });
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenCompose(
+ empty -> {
+ if (triggerResult.get().isPurge()) {
+ return windowState.asyncClear();
+ } else {
+ return StateFutureUtils.completedFuture(null);
+ }
+ })
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
+ }
+
+ // side output input event if
+ // element not handled by any window
+ // late arriving tag has been set
+ // windowAssigner is event time and current timestamp + allowed lateness no less than
+ // element timestamp
+ StateFutureUtils.combineAll(windowFutures)
+ .thenAccept(
+ isSkippedElements -> {
+ boolean isSkippedElement = true;
+ for (Boolean isSkipped : isSkippedElements) {
+ if (!isSkipped) {
+ isSkippedElement = false;
+ break;
+ }
+ }
+
+ if (isSkippedElement && isElementLate(element)) {
+ if (lateDataOutputTag != null) {
+ sideOutput(element);
+ } else {
+ this.numLateRecordsDropped.inc();
+ }
+ }
+ });
+ }
+ }
+
+ @Override
+ public void onEventTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onEventTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public void onProcessingTime(InternalTimer<K, W> timer) throws Exception {
+ keyDeclaredVariable.set(timer.getKey());
+ windowDeclaredVariable.set(timer.getNamespace());
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ triggerContext
+ .onProcessingTime(timer.getTimestamp())
+ .thenAccept(triggerResult::set)
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isFire()
+ ? windowState
+ .asyncGet()
+ .thenCompose(
+ contents ->
+ contents != null
+ ? emitWindowContents(
+ timer.getKey(),
+ timer
+ .getNamespace(),
+ contents)
+ : StateFutureUtils
+ .completedFuture(
+ null))
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ ignore ->
+ triggerResult.get().isPurge()
+ ? windowState.asyncClear()
+ : StateFutureUtils.completedFuture(null))
+ .thenCompose(
+ empty ->
+ (!windowAssigner.isEventTime()
+ && isCleanupTime(
+ timer.getNamespace(),
+ timer.getTimestamp()))
+ ? clearAllState(timer.getNamespace(), windowState)
+ : StateFutureUtils.completedFuture(null));
+ }
+ }
+
+ @Override
+ public OperatorAttributes getOperatorAttributes() {
+ boolean isOutputOnlyAfterEndOfStream =
+ windowAssigner instanceof GlobalWindows
+ && trigger instanceof AsyncEndOfStreamTrigger;
+ return new OperatorAttributesBuilder()
+ .setOutputOnlyAfterEndOfStream(isOutputOnlyAfterEndOfStream)
+ .build();
+ }
+
+ /**
+ * Drops all state for the given window and calls {@link AsyncTrigger#clear(Window,
+ * AsyncTrigger.TriggerContext)}.
+ *
+ * <p>The caller must ensure that the correct key is set in the state backend and the
+ * triggerContext object.
+ */
+ private StateFuture<Void> clearAllState(W window, AppendingState<IN, ACC, ACC> windowState) {
+ return windowState
+ .asyncClear()
+ .thenCompose(ignore -> triggerContext.clear())
+ .thenCompose(
+ ignore -> {
+ windowDeclaredVariable.set(window);
+ return processContext.clear();
+ });
+ }
+
+ /** Emits the contents of the given window using the {@link InternalWindowFunction}. */
+ @SuppressWarnings("unchecked")
+ private StateFuture<Void> emitWindowContents(K key, W window, ACC contents) throws Exception {
+ timestampedCollector.setAbsoluteTimestamp(window.maxTimestamp());
+ windowDeclaredVariable.set(window);
+ return userFunction.process(key, window, processContext, contents, timestampedCollector);
+ }
+
+ /**
+ * Write skipped late arriving element to SideOutput.
+ *
+ * @param element skipped late arriving element to side output
+ */
+ protected void sideOutput(StreamRecord<IN> element) {
+ output.collect(lateDataOutputTag, element);
+ }
+
+ /**
+ * Returns {@code true} if the watermark is after the end timestamp plus the allowed lateness of
+ * the given window.
+ */
+ protected boolean isWindowLate(W window) {
+ return (windowAssigner.isEventTime()
+ && (cleanupTime(window) <= internalTimerService.currentWatermark()));
+ }
+
+ /**
+ * Decide if a record is currently late, based on current watermark and allowed lateness.
+ *
+ * @param element The element to check
+ * @return The element for which should be considered when sideoutputs
+ */
+ protected boolean isElementLate(StreamRecord<IN> element) {
+ return (windowAssigner.isEventTime())
+ && (element.getTimestamp() + allowedLateness
+ <= internalTimerService.currentWatermark());
+ }
+
+ /**
+ * Registers a timer to cleanup the content of the window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void registerCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // don't set a GC timer for "end of time"
+ return;
+ }
+
+ if (windowAssigner.isEventTime()) {
+ triggerContext.registerEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.registerProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Deletes the cleanup timer set for the contents of the provided window.
+ *
+ * @param window the window whose state to discard
+ */
+ protected void deleteCleanupTimer(W window) {
+ long cleanupTime = cleanupTime(window);
+ if (cleanupTime == Long.MAX_VALUE) {
+ // no need to clean up because we didn't set one
+ return;
+ }
+ if (windowAssigner.isEventTime()) {
+ triggerContext.deleteEventTimeTimer(cleanupTime);
+ } else {
+ triggerContext.deleteProcessingTimeTimer(cleanupTime);
+ }
+ }
+
+ /**
+ * Returns the cleanup time for a window, which is {@code window.maxTimestamp +
+ * allowedLateness}. In case this leads to a value greater than {@link Long#MAX_VALUE} then a
+ * cleanup time of {@link Long#MAX_VALUE} is returned.
+ *
+ * @param window the window whose cleanup time we are computing.
+ */
+ private long cleanupTime(W window) {
+ if (windowAssigner.isEventTime()) {
+ long cleanupTime = window.maxTimestamp() + allowedLateness;
+ return cleanupTime >= window.maxTimestamp() ? cleanupTime : Long.MAX_VALUE;
+ } else {
+ return window.maxTimestamp();
+ }
+ }
+
+ /** Returns {@code true} if the given time is the cleanup time for the given window. */
+ protected final boolean isCleanupTime(W window, long time) {
+ return time == cleanupTime(window);
+ }
+
+ /**
+ * Base class for per-window {@link KeyedStateStore KeyedStateStores}. Used to allow per-window
+ * state access for {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public abstract class AbstractPerWindowStateStore extends DefaultKeyedStateStore {
+
+ // we have this in the base class even though it's not used in MergingKeyStore so that
+ // we can always set it and ignore what actual implementation we have
+ protected final DeclaredVariable<W> window;
+
+ public AbstractPerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(
+ keyedStateBackend,
+ new SerializerFactory() {
+ @Override
+ public <T> TypeSerializer<T> createSerializer(
+ TypeInformation<T> typeInformation) {
+ return typeInformation.createSerializer(
+ executionConfig.getSerializerConfig());
+ }
+ });
+ this.window = window;
+ }
+ }
+
+ /**
+ * Regular per-window state store for use with {@link
+ * org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction}.
+ */
+ public class PerWindowStateStore extends AbstractPerWindowStateStore {
+
+ public PerWindowStateStore(
+ AsyncKeyedStateBackend<?> keyedStateBackend,
+ ExecutionConfig executionConfig,
+ DeclaredVariable<W> window) {
+ super(keyedStateBackend, executionConfig, window);
+ }
+
+ @Override
+ protected <S extends State, SV> S getPartitionedState(StateDescriptor<SV> stateDescriptor)
+ throws Exception {
+ return stateWrapWithNamespace(
+ asyncKeyedStateBackend.getOrCreateKeyedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ }
+ }
+
+ /**
+ * A utility class for handling {@code ProcessWindowFunction} invocations. This can be reused by
+ * setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code WindowContext}.
+ */
+ public class WindowContext implements InternalAsyncWindowFunction.InternalWindowContext {
+ protected final DeclaredVariable<W> window;
+
+ protected AbstractPerWindowStateStore windowState;
+
+ public WindowContext(DeclaredVariable<W> window) {
+ this.window = window;
+ // TODO: support merging window.
+ this.windowState =
+ new PerWindowStateStore(
+ getAsyncKeyedStateBackend(), getExecutionConfig(), window);
+ }
+
+ @Override
+ public String toString() {
+ return "WindowContext{Window = " + window.toString() + "}";
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return userFunction.clear(window.get(), this);
+ }
+
+ @Override
+ public long currentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public long currentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @Override
+ public KeyedStateStore windowState() {
+ return this.windowState;
+ }
+
+ @Override
+ public KeyedStateStore globalState() {
+ return AsyncWindowOperator.this.getKeyedStateStore();
+ }
+
+ @Override
+ public <X> void output(OutputTag<X> outputTag, X value) {
+ if (outputTag == null) {
+ throw new IllegalArgumentException("OutputTag must not be null.");
+ }
+ output.collect(outputTag, new StreamRecord<>(value, window.get().maxTimestamp()));
+ }
+ }
+
+ /**
+ * {@code Context} is a utility for handling {@code AsyncTrigger} invocations. It can be reused
+ * by setting the {@code key} and {@code window} fields. No internal state must be kept in the
+ * {@code Context}
+ */
+ public class Context implements AsyncTrigger.OnMergeContext {
+ protected DeclaredVariable<K> key;
+ protected DeclaredVariable<W> window;
+
+ protected Collection<W> mergedWindows;
+
+ public Context(DeclaredVariable<K> key, DeclaredVariable<W> window) {
+ this.key = key;
+ this.window = window;
+ }
+
+ @Override
+ public MetricGroup getMetricGroup() {
+ return AsyncWindowOperator.this.getMetricGroup();
+ }
+
+ public long getCurrentWatermark() {
+ return internalTimerService.currentWatermark();
+ }
+
+ @SuppressWarnings("unchecked")
+ public <T, S extends State> S getPartitionedState(StateDescriptor<T> stateDescriptor) {
+ try {
+ return stateWrapWithNamespace(
+ AsyncWindowOperator.this.getAsyncPartitionedState(
+ window.get(), windowSerializer, stateDescriptor),
+ window);
+ } catch (Exception e) {
+ throw new RuntimeException("Could not retrieve state", e);
+ }
+ }
+
+ @Override
+ public <T> void mergePartitionedState(StateDescriptor<T> stateDescriptor) {
+ throw new UnsupportedOperationException("Merging window not supported yet");
+ }
+
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+
+ @Override
+ public void registerProcessingTimeTimer(long time) {
+ internalTimerService.registerProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void registerEventTimeTimer(long time) {
+ internalTimerService.registerEventTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteProcessingTimeTimer(long time) {
+ internalTimerService.deleteProcessingTimeTimer(window.get(), time);
+ }
+
+ @Override
+ public void deleteEventTimeTimer(long time) {
+ internalTimerService.deleteEventTimeTimer(window.get(), time);
+ }
+
+ public StateFuture<TriggerResult> onElement(StreamRecord<IN> element) throws Exception {
+ return trigger.onElement(
+ element.getValue(), element.getTimestamp(), window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onProcessingTime(long time) throws Exception {
+ return trigger.onProcessingTime(time, window.get(), this);
+ }
+
+ public StateFuture<TriggerResult> onEventTime(long time) throws Exception {
+ return trigger.onEventTime(time, window.get(), this);
+ }
+
+ public void onMerge(Collection<W> mergedWindows) throws Exception {
+ throw new UnsupportedOperationException("Merging window not support yet");
+ }
+
+ public StateFuture<Void> clear() throws Exception {
+ return trigger.clear(window.get(), this);
+ }
+
+ @Override
+ public String toString() {
+ return "Context{" + "key=" + key + ", window=" + window + '}';
+ }
+ }
+
+ /** Internal class for keeping track of in-flight timers. */
+ protected static class Timer<K, W extends Window> implements Comparable<Timer<K, W>> {
+ protected long timestamp;
+ protected K key;
+ protected W window;
+
+ public Timer(long timestamp, K key, W window) {
+ this.timestamp = timestamp;
+ this.key = key;
+ this.window = window;
+ }
+
+ @Override
+ public int compareTo(Timer<K, W> o) {
+ return Long.compare(this.timestamp, o.timestamp);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+
+ Timer<?, ?> timer = (Timer<?, ?>) o;
+
+ return timestamp == timer.timestamp
+ && key.equals(timer.key)
+ && window.equals(timer.window);
+ }
+
+ @Override
+ public int hashCode() {
+ int result = (int) (timestamp ^ (timestamp >>> 32));
+ result = 31 * result + key.hashCode();
+ result = 31 * result + window.hashCode();
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "Timer{"
+ + "timestamp="
+ + timestamp
+ + ", key="
+ + key
+ + ", window="
+ + window
+ + '}';
+ }
+ }
+
+ protected <S extends State, N, T> S getAsyncPartitionedState(
+ N namespace, TypeSerializer<N> namespaceSerializer, StateDescriptor<T> stateDescriptor)
+ throws Exception {
+ return getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(namespace, namespaceSerializer, stateDescriptor);
+ }
Review Comment:
Resolved, I just delete this useless function and let the caller directly use ` getAsyncKeyedStateBackend().getOrCreateKeyedState()`, See
```
Context#getPartitionedState(StateDescriptor<T> stateDescriptor)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920149771
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,810 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.TimestampedCollectorWithDeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.create;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollectorWithDeclaredVariable<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void open() throws Exception {
+ super.open();
+ Preconditions.checkNotNull(
+ getAsyncKeyedStateBackend(), "Async state backend is not properly set.");
+
+ this.numLateRecordsDropped = metrics.counter(LATE_ELEMENTS_DROPPED_METRIC_NAME);
+ timestampedCollector =
+ new TimestampedCollectorWithDeclaredVariable<>(output, declarationContext);
+
+ internalTimerService = getInternalTimerService("window-timers", windowSerializer, this);
+
+ windowDeclaredVariable =
+ declarationContext.declareVariable(
+ windowSerializer,
+ "_AsyncWindowOperator$window",
+ windowSerializer::createInstance);
+
+ triggerContext = new Context(windowDeclaredVariable);
+ processContext = new WindowContext(windowDeclaredVariable);
+
+ windowAssignerContext =
+ new WindowAssigner.WindowAssignerContext() {
+ @Override
+ public long getCurrentProcessingTime() {
+ return internalTimerService.currentProcessingTime();
+ }
+ };
+
+ // create (or restore) the state that hold the actual window contents
+ // NOTE - the state may be null in the case of the overriding evicting window operator
+ if (windowStateDescriptor != null) {
+ windowState =
+ create(
+ getAsyncKeyedStateBackend()
+ .getOrCreateKeyedState(
+ windowSerializer.createInstance(),
+ windowSerializer,
+ windowStateDescriptor),
+ windowDeclaredVariable);
+ }
+
+ // create the typed and helper states for merging windows
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ }
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ timestampedCollector = null;
+ triggerContext = null;
+ processContext = null;
+ windowAssignerContext = null;
+ }
+
+ @Override
+ public void processElement(StreamRecord<IN> element) throws Exception {
+ final Collection<W> elementWindows =
+ windowAssigner.assignWindows(
+ element.getValue(), element.getTimestamp(), windowAssignerContext);
+
+ final K key = (K) getCurrentKey();
+ Collection<StateFuture<Boolean>> windowFutures = new ArrayList<>();
+
+ if (windowAssigner instanceof MergingWindowAssigner) {
+ throw new UnsupportedOperationException(
+ "Async WindowOperator not support merging window (e.g. session window) yet.");
+ } else {
+ for (W window : elementWindows) {
+
+ // drop if the window is already late
+ if (isWindowLate(window)) {
+ continue;
+ }
+
+ AtomicReference<TriggerResult> triggerResult = new AtomicReference<>();
+ windowDeclaredVariable.set(window);
+
+ windowFutures.add(
+ windowState
+ .asyncAdd(element.getValue())
+ .thenCompose(
+ ignore ->
+ triggerContext
+ .onElement(element)
+ .thenAccept(triggerResult::set))
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isFire(),
+ ignore ->
+ windowState
+ .asyncGet()
+ .thenConditionallyCompose(
+ Objects::nonNull,
+ contents ->
+ emitWindowContents(
+ key, window,
+ contents),
+ StateFutureUtils
+ .createVoidReturnFunction()),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenConditionallyCompose(
+ ignore -> triggerResult.get().isPurge(),
+ ignore -> windowState.asyncClear(),
+ StateFutureUtils.createVoidReturnFunction())
+ .thenApply(
+ empty -> {
+ registerCleanupTimer(window);
+ // if element is handled by none of assigned
+ // elementWindows
+ return false; // isSkippedElement = false
+ }));
Review Comment:
Thank you for your reminding. I have use `windowFutures` to implicitly indicate whether `isSkippedElement == true`, i.e., `windowFutures is empty => implicitly isSkippedElement == true`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920944894
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/triggers/AsyncTrigger.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+
+import java.io.Serializable;
+
+/**
+ * A {@code Trigger} determines when a pane of a window should be evaluated to emit the results for
+ * that part of the window. This trigger is for the async window operator, i.e., {@link
+ * org.apache.flink.runtime.asyncprocessing.operators.windowing.AsyncWindowOperator}.
+ *
+ * <p>All State APIs in the async trigger are from State V2.
+ *
+ * <p>A pane is the bucket of elements that have the same key (assigned by the {@link
+ * org.apache.flink.api.java.functions.KeySelector}) and same {@link Window}. An element can be in
+ * multiple panes if it was assigned to multiple windows by the {@link
+ * org.apache.flink.streaming.api.windowing.assigners.WindowAssigner}. These panes all have their
+ * own instance of the {@code Trigger}.
+ *
+ * <p>Triggers must not maintain state internally since they can be re-created or reused for
+ * different keys. All necessary state should be persisted using the state abstraction available on
+ * the {@link TriggerContext}.
+ *
+ * <p>When used with a {@link
+ * org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner} the {@code Trigger}
+ * must return {@code true} from {@link #canMerge()} and {@link #onMerge(Window, OnMergeContext)}
+ * most be properly implemented.
+ *
+ * @param <T> The type of elements on which this {@code Trigger} works.
+ * @param <W> The type of {@link Window Windows} on which this {@code Trigger} can operate.
+ */
+@Experimental
Review Comment:
We'd better mark all the async triggers as `@Internal` since they are not exposed to users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1917933066
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/ValueStateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/** Value state wrapped with declared namespace. */
+@Internal
+class ValueStateWithDeclaredNamespace<K, N, V> extends StateWithDeclaredNamespace<N>
+ implements InternalValueState<K, N, V> {
+ private final InternalValueState<K, N, V> state;
+
+ public ValueStateWithDeclaredNamespace(
+ InternalValueState<K, N, V> state, DeclaredVariable<N> declaredNamespace) {
+ super(state, declaredNamespace);
+ this.state = state;
+ }
+
+ @Override
+ public StateFuture<Void> asyncClear() {
+ wrapNamespace();
+ return state.asyncClear();
+ }
+
+ @Override
+ public StateFuture<V> asyncValue() {
+ wrapNamespace();
+ return state.asyncValue();
+ }
+
+ @Override
+ public StateFuture<Void> asyncUpdate(V value) {
+ wrapNamespace();
+ return state.asyncUpdate(value);
+ }
+
+ @Override
+ public void clear() {
+ state.clear();
+ }
+
+ @Override
+ public V value() {
+ return state.value();
+ }
+
+ @Override
+ public void update(V value) {
+ state.update(value);
+ }
+
+ @Override
+ public void setCurrentNamespace(N namespace) {
+ state.setCurrentNamespace(namespace);
Review Comment:
The declared namespace should also be set here. Then the `mergeNamespaces` will work properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920952684
##########
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers/AsyncContinuousEventTimeTrigger.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.windowing.triggers;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.common.state.v2.ReducingState;
+import org.apache.flink.api.common.state.v2.ReducingStateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeutils.base.LongSerializer;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+
+import java.time.Duration;
+
+/**
+ * A {@link Trigger} that continuously fires based on a given time interval. This fires based on
+ * {@link org.apache.flink.streaming.api.watermark.Watermark Watermarks}.
+ *
+ * @param <W> The type of {@link Window Windows} on which this trigger can operate.
+ * @see org.apache.flink.streaming.api.watermark.Watermark
+ */
+@PublicEvolving
+public class AsyncContinuousEventTimeTrigger<W extends Window> extends AsyncTrigger<Object, W> {
Review Comment:
IIUC, this is not actually used. Can we move this to the package `org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers` and mark as `@Internal`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on PR #25996:
URL: https://github.com/apache/flink/pull/25996#issuecomment-2599534361
> LGTM. It's better to parameterize the test also.
@Zakelly Thanks for reviewing. I have refactor the UT of window. After refactoring, existing window UT (i.e., `WindowOperatorTest` and `EvictingWindowOperatorTest`) can run with paramater `enableAsyncState`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918042057
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/operators/windowing/AsyncWindowOperator.java:
##########
@@ -0,0 +1,902 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.operators.windowing;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.functions.SerializerFactory;
+import org.apache.flink.api.common.state.KeyedStateStore;
+import org.apache.flink.api.common.state.v2.AppendingState;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.api.common.state.v2.StateDescriptor;
+import org.apache.flink.api.common.state.v2.StateFuture;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.core.state.StateFutureUtils;
+import org.apache.flink.metrics.Counter;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.asyncprocessing.operators.AbstractAsyncStateUdfStreamOperator;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.functions.InternalAsyncWindowFunction;
+import org.apache.flink.runtime.asyncprocessing.operators.windowing.triggers.AsyncTrigger;
+import org.apache.flink.runtime.state.AsyncKeyedStateBackend;
+import org.apache.flink.runtime.state.DefaultKeyedStateStore;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.v2.internal.InternalAppendingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMergingState;
+import org.apache.flink.streaming.api.graph.StreamConfig;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.InternalTimerService;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.api.operators.OperatorAttributes;
+import org.apache.flink.streaming.api.operators.OperatorAttributesBuilder;
+import org.apache.flink.streaming.api.operators.Output;
+import org.apache.flink.streaming.api.operators.TimestampedCollector;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows;
+import org.apache.flink.streaming.api.windowing.assigners.GlobalWindows.AsyncEndOfStreamTrigger;
+import org.apache.flink.streaming.api.windowing.assigners.MergingWindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner;
+import org.apache.flink.streaming.api.windowing.assigners.WindowAssigner.WindowAssignerContext;
+import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
+import org.apache.flink.streaming.api.windowing.windows.Window;
+import org.apache.flink.streaming.runtime.operators.windowing.functions.InternalWindowFunction;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.runtime.tasks.ProcessingTimeService;
+import org.apache.flink.streaming.runtime.tasks.StreamTask;
+import org.apache.flink.util.OutputTag;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicReference;
+
+import static org.apache.flink.runtime.asyncprocessing.declare.state.StateWithDeclaredNamespace.stateWrapWithNamespace;
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * An operator that implements the logic for windowing based on a {@link WindowAssigner} and {@link
+ * AsyncTrigger}. This is the async state access version of WindowOperator.
+ *
+ * <p>When an element arrives it gets assigned a key using a {@link KeySelector} and it gets
+ * assigned to zero or more windows using a {@link WindowAssigner}. Based on this, the element is
+ * put into panes. A pane is the bucket of elements that have the same key and same {@code Window}.
+ * An element can be in multiple panes if it was assigned to multiple windows by the {@code
+ * WindowAssigner}.
+ *
+ * <p>Each pane gets its own instance of the provided {@code Trigger}. This trigger determines when
+ * the contents of the pane should be processed to emit results. When a trigger fires, the given
+ * {@link InternalWindowFunction} is invoked to produce the results that are emitted for the pane to
+ * which the {@code Trigger} belongs.
+ *
+ * @param <K> The type of key returned by the {@code KeySelector}.
+ * @param <IN> The type of the incoming elements.
+ * @param <OUT> The type of elements emitted by the {@code InternalWindowFunction}.
+ * @param <W> The type of {@code Window} that the {@code WindowAssigner} assigns.
+ */
+@Internal
+public class AsyncWindowOperator<K, IN, ACC, OUT, W extends Window>
+ extends AbstractAsyncStateUdfStreamOperator<
+ OUT, InternalAsyncWindowFunction<ACC, OUT, K, W>>
+ implements OneInputStreamOperator<IN, OUT>, Triggerable<K, W> {
+
+ private static final long serialVersionUID = 1L;
+
+ private static final Logger LOG = LoggerFactory.getLogger(AsyncWindowOperator.class);
+
+ // ------------------------------------------------------------------------
+ // Configuration values and user functions
+ // ------------------------------------------------------------------------
+
+ protected final WindowAssigner<? super IN, W> windowAssigner;
+
+ private final KeySelector<IN, K> keySelector;
+
+ private final AsyncTrigger<? super IN, ? super W> trigger;
+
+ private final StateDescriptor<?> windowStateDescriptor;
+
+ /** For serializing the key in checkpoints. */
+ protected final TypeSerializer<K> keySerializer;
+
+ /** For serializing the window in checkpoints. */
+ protected final TypeSerializer<W> windowSerializer;
+
+ /**
+ * The allowed lateness for elements. This is used for:
+ *
+ * <ul>
+ * <li>Deciding if an element should be dropped from a window due to lateness.
+ * <li>Clearing the state of a window if the system time passes the {@code window.maxTimestamp
+ * + allowedLateness} landmark.
+ * </ul>
+ */
+ protected final long allowedLateness;
+
+ /**
+ * {@link OutputTag} to use for late arriving events. Elements for which {@code
+ * window.maxTimestamp + allowedLateness} is smaller than the current watermark will be emitted
+ * to this.
+ */
+ protected final OutputTag<IN> lateDataOutputTag;
+
+ private static final String LATE_ELEMENTS_DROPPED_METRIC_NAME = "numLateRecordsDropped";
+
+ protected transient Counter numLateRecordsDropped;
+
+ // ------------------------------------------------------------------------
+ // State that is not checkpointed
+ // ------------------------------------------------------------------------
+
+ /** The state in which the window contents is stored. Each window is a namespace */
+ private transient InternalAppendingState<K, W, IN, ACC, ACC, ACC> windowState;
+
+ /**
+ * The {@link #windowState}, typed to merging state for merging windows. Null if the window
+ * state is not mergeable. TODO: Not support currently.
+ */
+ private transient InternalMergingState<K, W, IN, ACC, ACC, ACC> windowMergingState;
+
+ /**
+ * The state that holds the merging window metadata (the sets that describe what is merged).
+ * TODO: Not support currently.
+ */
+ private transient InternalListState<K, VoidNamespace, Tuple2<W, W>> mergingSetsState;
+
+ /**
+ * This is given to the {@code InternalWindowFunction} for emitting elements with a given
+ * timestamp.
+ */
+ protected transient TimestampedCollector<OUT> timestampedCollector;
+
+ protected transient Context triggerContext;
+
+ protected transient WindowContext processContext;
+
+ protected transient DeclaredVariable<W> windowDeclaredVariable;
+ protected transient DeclaredVariable<K> keyDeclaredVariable;
+
+ protected transient WindowAssigner.WindowAssignerContext windowAssignerContext;
+
+ // ------------------------------------------------------------------------
+ // State that needs to be checkpointed
+ // ------------------------------------------------------------------------
+
+ protected transient InternalTimerService<W> internalTimerService;
+
+ /** Creates a new {@code WindowOperator} based on the given policies and user functions. */
+ public AsyncWindowOperator(
+ WindowAssigner<? super IN, W> windowAssigner,
+ TypeSerializer<W> windowSerializer,
+ KeySelector<IN, K> keySelector,
+ TypeSerializer<K> keySerializer,
+ StateDescriptor<?> windowStateDescriptor,
+ InternalAsyncWindowFunction<ACC, OUT, K, W> windowFunction,
+ AsyncTrigger<? super IN, ? super W> trigger,
+ long allowedLateness,
+ OutputTag<IN> lateDataOutputTag) {
+
+ super(windowFunction);
+
+ checkArgument(allowedLateness >= 0);
+
+ checkArgument(
+ windowStateDescriptor == null || windowStateDescriptor.isSerializerInitialized(),
+ "window state serializer is not properly initialized");
+
+ this.windowAssigner = checkNotNull(windowAssigner);
+ this.windowSerializer = checkNotNull(windowSerializer);
+ this.keySelector = checkNotNull(keySelector);
+ this.keySerializer = checkNotNull(keySerializer);
+ this.windowStateDescriptor = windowStateDescriptor;
+ this.trigger = checkNotNull(trigger);
+ this.allowedLateness = allowedLateness;
+ this.lateDataOutputTag = lateDataOutputTag;
+
+ LOG.info("Initialize async window operator with trigger " + trigger);
+ }
+
+ @Override
+ public void setup(
+ StreamTask<?, ?> containingTask,
+ StreamConfig config,
+ Output<StreamRecord<OUT>> output) {
+ super.setup(containingTask, config, output);
+ }
+
+ @Override
+ public void setProcessingTimeService(ProcessingTimeService processingTimeService) {
+ super.setProcessingTimeService(processingTimeService);
+ }
Review Comment:
Seems no need to override.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1918320158
##########
flink-runtime/src/main/java/org/apache/flink/runtime/asyncprocessing/declare/state/StateWithDeclaredNamespace.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.asyncprocessing.declare.state;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.v2.State;
+import org.apache.flink.runtime.asyncprocessing.declare.DeclaredVariable;
+import org.apache.flink.runtime.state.v2.internal.InternalAggregatingState;
+import org.apache.flink.runtime.state.v2.internal.InternalListState;
+import org.apache.flink.runtime.state.v2.internal.InternalMapState;
+import org.apache.flink.runtime.state.v2.internal.InternalPartitionedState;
+import org.apache.flink.runtime.state.v2.internal.InternalReducingState;
+import org.apache.flink.runtime.state.v2.internal.InternalValueState;
+
+/**
+ * State wrapped by declared namespace. This is useful for window operator, where we need to
+ * frequently setup current namespace in each lambda. This wrap class help use to invoke {@code
+ * setCurrentNamespace} when we are going to call async state API.
+ */
+@Internal
+public class StateWithDeclaredNamespace<N> {
+ private final InternalPartitionedState<N> state;
+ private final DeclaredVariable<N> declaredNamespace;
+
+ public StateWithDeclaredNamespace(State state, DeclaredVariable<N> declaredNamespace) {
+ if (state instanceof InternalPartitionedState) {
+ this.state = (InternalPartitionedState<N>) state;
+ } else {
+ this.state = null;
+ }
+ this.declaredNamespace = declaredNamespace;
+ }
+
+ protected void wrapNamespace() {
+ if (this.state != null) {
+ state.setCurrentNamespace(declaredNamespace.get());
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ public static <N, S extends State> S stateWrapWithNamespace(
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "jiexray (via GitHub)" <gi...@apache.org>.
jiexray commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1920130658
##########
flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/WindowWordCount.java:
##########
@@ -118,15 +129,33 @@ public static void main(String[] args) throws Exception {
// Using a keyBy allows performing aggregations and other
// stateful transformations over data on a per-key basis.
// This is similar to a GROUP BY clause in a SQL query.
- .keyBy(value -> value.f0)
- // create windows of windowSize records slided every slideSize records
- .countWindow(windowSize, slideSize)
- // For each key, we perform a simple sum of the "1" field, the count.
- // If the input data set is bounded, sum will output a final count for
- // each word. If it is unbounded, it will continuously output updates
- // each time it sees a new instance of each word in the stream.
- .sum(1)
- .name("counter");
+ .keyBy(value -> value.f0);
+
+ DataStream<Tuple2<String, Integer>> counts;
+ if (params.isAsyncState()) {
+ counts =
+ keyedStream
+ .enableAsyncState()
+ // create windows of windowSize records slided every slideSize records
+ .asyncCountWindow(windowSize, slideSize)
Review Comment:
Resolved, I have removed this API. countWindow() + enableAsyncState() will trigger DataStream using AsyncWindowOperator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1921497698
##########
flink-runtime/src/main/java/org/apache/flink/streaming/api/transformations/OneInputTransformation.java:
##########
@@ -192,4 +192,9 @@ public boolean isOutputOnlyAfterEndOfStream() {
public boolean isInternalSorterSupported() {
return operatorFactory.getOperatorAttributes().isInternalSorterSupported();
}
+
+ @Override
+ public void enableAsyncState() {
+ // nothing to do.
+ }
Review Comment:
Here we should apply the change like [this PR](https://github.com/apache/flink/pull/26000/files#diff-a3035df99b00d9ab9a00e5050b2377d6cbb9ab7db0766ec9d88dc504a2643cfeR189-R198). Or after that is merged, rebase on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37028] Datastream Window operator with async state processing [flink]
Posted by "Zakelly (via GitHub)" <gi...@apache.org>.
Zakelly commented on code in PR #25996:
URL: https://github.com/apache/flink/pull/25996#discussion_r1919492172
##########
flink-core/src/main/java/org/apache/flink/core/state/StateFutureUtils.java:
##########
@@ -34,6 +35,14 @@
*/
@Experimental
public class StateFutureUtils {
+ /**
+ * Returns a function that always returns a completed future that does nothing and return null.
+ */
+ public static <T>
+ FunctionWithException<T, StateFuture<T>, Exception> createVoidReturnFunction() {
Review Comment:
```suggestion
FunctionWithException<T, StateFuture<Void>, Exception> returnVoidFuture() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org